How to set up your local projects to contribute to Laravel (or any PHP library) : a practical guide

Let’s imagine this scenario: You are working on a Laravel application, and you found a bug either in the framework itself or in one of the third party libraries you are using. You edit the code inside your project, or maybe you installed a fresh copy of Laravel and library that has the issue, and you edited the code there. You saved the changes, but when you went to your terminal to commit the changes, GIT just ignored what you did. You start wondering “What’s happening here?”.

 

The first solution that might pop up in your head could be to read the “How to contribute” of the framework or the library documentation, and in most cases, you’d find there the necessary steps to contribute to the framework, but I always felt that something was missing in the documentation. Let me explain what I mean by that:

Let’s take the example of Laravel here, if you decide to contribute to the framework, you’d need to clone the laravel/framework repository, push your changes to your own copy of the repository (on github), and create a PR, but what’s missing here is How would you test your changes (bug fix, new feature, …) in a local project, since if you introduce the changes in that project you just can’t commit them and create a PR for them easily, and if you change the code in the laravel/framework repository you just cloned you might end up doing a lot of copy/pasting between this repository and your project in order to test those changes.

Luckily there is a cleaner way to deal with all this. I’m not going to walk you through the usual steps of creating a PR, I assume since you are reading this article you are already familiar with that. I’ll focus on how you should set up your local environment to streamline this whole process and show you how you could fix bugs and develop new features in the framework/library and test them locally before you create the PR you are about to send.

Prepare your test project and your repository

First, you’d need to fork the laravel/framework repository and then clone it locally (the usual steps when it comes to contributing to any open source project) and then create a test application where you are going to test those changes (I’m calling it playground)

➜ laravel pwd
/home/djug/laravel
➜ laravel ls
framework playground

Make Composer load the changes from the local repository

Now, with the most important part. We need to tell the playground application to take into consideration the changes we introduce in the /home/djug/laravel/framework repository

We can do this via composer, and according to its documentation and I’m quoting here:

Loading a package from a VCS repository

There are a few use cases for this. The most common one is maintaining
your own fork of a third party library. If you are using a certain
library for your project and you decide to change something in the
library, you will want your project to use the patched version. If the
library is on GitHub (this is the case most of the time), you can
simply fork it there and push your changes to your fork. After that,
you update the project’s composer.json. All you have to do is add your
fork as a repository and update the version constraint to point to
your custom branch.

So the first part of the composer.json file (on my playground project) should look like this:

{
    "name": "laravel/laravel",
    "type": "project",
    "description": "The Laravel Framework.",
    "keywords": [
    "framework",
    "laravel"
    ],

    "license": "MIT",
    "repositories": [
        {
            "type": "vcs",
            "url": "/home/djug/laravel/framework/"
        }
    ],
    "require": {
        "php": "^7.1.3",
        "fideloper/proxy": "^4.0",
        "laravel/framework": "master",
        "laravel/tinker": "^1.0"
    },
...
}

Now each time I change something in the /home/djug/laravel/framework/ repository and commit the change (you need to commit it) and if I execute composer update on my playground project it will get the changes.

3/ Push the changes to your remote repository and create a PR

Now that your playground repository can pull the changes from your framework repository, you could update add all the changes you want to your code and then test them on a “real” project.

After you finish testing, all that you need to do is to push the changes to your remote repository (your own “copy” of the framework repository on GitHub for instance) and then create the pull request.

 

Photo by John Schnobrich on Unsplash

Adding Tests to your Laravel CRUD Application: Where to Start?

This is an excerpt from my upcoming ebook Laravel testing 101

Imagine you are working on a Laravel side-project or you just joined a team working on a Laravel application, and it currently doesn’t have a single test. You want to change that, and add tests to the application. If you find yourself in this situation, you might not even know where to start. “Should I unit test everything? Should I just test the most critical parts of the application?” You might even find yourself asking “Why do I need tests in the first place?

I am writing an ebook to answer all these questions for you, and this is the introductory chapter. My main goal with this ebook  is to try to answer these questions, and I will describe my process in writing tests for some of the Laravel applications I’m working on (mainly CRUD applications).

 

What you are going to read in this chapter and the following ones might not necessarily be the optimal way of testing or even the best practice, but rather just one possible way of doing testing in Laravel [CRUD] applications, the way I am approaching testing these days.

 

Why we are writing fewer tests than necessary

For me, testing is just like physical exercise. We all know that we should be exercising (or exercising more), but very few of us exercise on a regular basis, if at all.

Many people decide they’ll be exercising every single day as part of their new year resolution. They get a yearly subscription at their local gym to prove how serious they are, they might even hit the gym 5 days in a row the first week of the year. But after the first weekend, they start going less and less, and end up just giving up because their body aches so much and they don’t see any improvement yet (they worked out for a WHOLE week, they should at least notice a quick drop in their weights, no?). The other scenario is that they might feel overwhelmed with all the choices they find at the gym: Should I focus just on cardio? Maybe lift some weights? Or maybe I should do both? Maybe I’ll just focus on my biceps… A classic case of paralysis by analysis.

Almost the same thing happens with developers and testing.

With every new project, they decide they’ll unit test the sh*t out of it. They decide to test every single class and method (even the private ones), they would even buy some ebooks or online courses to learn how to implement a testing approach in their project.

They’ll start implementing TDD, for instance, in the first week of the project, but after the first weekend passes by, they’ll start thinking about giving up. Not only they don’t see any improvement in their development process or in the end results (they have been TDDing everything for a whole week for God’s sake), but also they’ll start writing tests for just a method or two on each class, and then start to question if this approach is the right one or if they should try something else entirely, and then, the inevitable happens, they drop everything.

In other words, testing is just like exercising, we tend to think that we should do it perfectly (exercising every single day), or we shouldn’t bother at all. When in reality, all what we need is some good practices, and a change in the way we view and approach testing. We have to understand why we need tests in the first place, and learn how we can get the most out of them without doing too much.

 

Why do we need tests in the first place?

If you ask this question to a group of developers, you might not get the same answer twice. Personally, I write tests for one simple reason: to make sure that the application will keep working in the exact same way it was intended to when each piece of it was written. In other words, each time I introduce a new change to the code base or add new feature, I want to make sure that I’m not breaking any current functionalities.

If you think about it, each time you introduce a change and you start testing your app in the browser, you are just checking that everything is still working.  The method I’m describing here will just help you automate this whole process, i.e instead of opening random pages in the app to see whether they are still working, you execute your test suite, and confirm that everything is still working as expected.

Let me repeat it one more time, since this is really important, and all what follows is built upon this idea: write tests to make sure that the current functionalities keep working as expected when we introduce changes

In other words, testing should not be about proving that the code is 100% correct, all we need is to make sure that we are not going to break it or change its behavior in the future.

So, what should I be testing exactly?

As you can probably tell from the description above, we should be testing only controllers in this phase. If you think about it, if all the controllers are behaving exactly how they are supposed to, it is most likely that all the underlying code of the application (models, repositories, services…) is working just fine, unless your test suite doesn’t cover all the use cases of the controllers.

Structure of the ebook

What I don’t like in the majority of online resource about testing, is that they present each idea/concept in isolation. When usually, especially for a beginner, it is hard to tell how all these concepts fit together.

So instead of doing this, I’ll be adding tests to a small demo application I built for the purpose of this ebook.

It is a small application that allows users to sign up, and post articles. Each user can edit and delete their own articles, and read what other users post on the website.

The application is simple, but as you’ll see in the upcoming chapters, it contains enough code/functionalities to demonstrate the all the necessary ideas you would need in testing almost any Laravel CRUD application.

If you are interested, and you want to follow along, please visit laraveltesting101.com and sign up. You’ll be notified every time I publish something new.

 

laraveltesting101.com

[Laravel tips] How to temporarily change a laravel environment variable without updating the .env file

Imagine the following scenario: you are working on a laravel application that dispatches some jobs to a queue. You notice in your log file that a specific job is throwing an exception, but you can’t debug it directly (with dd() for instance), since the job is consumed by a worker elsewhere.

One way to debug this issue would be to change the QUEUE_DRIVER environment variable in your .env file to sync, debug, and then revert back the change after you finish.

Or imagine that you have a test database and you need to run the migrations on it first, and you’d need to update the .env file to use the test database first, run the migrations and then revert back the change.

As you can see, in both cases, all what we need to do is to update the environment variable temporarily. And usually, it takes some time to update the .env file (you might even need to clear the configuration or restart some workers).

Luckily, it is possible to pass an environment variable to linux commands, and these variables will only be used for the current execution.

In the previous example, all what we have to do is to execute the command as follows:

QUEUE_DRIVER=sync php artisan my-artisan-command-here

You can test it yourself with tinker.

You can even pass multiple environment variables like this:

QUEUE_DRIVER=sync DB_DATABASE=testing php artisan my-artisan-command-here

As you can see, this is not a laravel specific trick, and you can use it whenever you find yourself in a situation where you need to change an environment variable temporarily.

I hope you’ll find this useful.

[PoC] Partially random passwords: or how to protect users passwords from keyloggers with partially random passwords

One of the issues we all face when we login to some online accounts especially on public computers, or on any computer that we do not own, is that there is always a risk to get our passwords stolen especially with keyloggers. If a hacker gets a “copy” of your password, she can log in to your account and do whatever she wants.

As always, using 2 factor authentication can mitigate this issue, since the hacker needs to access your phone as well. But what if using 2FA is not an option, and we want to protect the user even in this case, can we detect,when the hacker tries to login, that it is not the real user but rather somebody who stole her password who is trying to log in?

Partially-Random passwords

One solution for this problem is to use partially-random passwords with a random part that is different each time.
What I mean by that is the following:
let’s assume that my password is 123456789, but when I try to login I will not use this password directly, but rather I’d add some random string to the beginning or to the end (or even in the middle) like this:
564564123456789

and when we receive this password (let’s call it the “raw password”), we will strip away the random part and use the remaining part as a password, and we save a hashed version of this raw password in a used_passwords table. And next time the same user tries to login, we check if this raw password (the real password + the random part) was used before, and we deny access to the account if we find it.
The code will look like this:

public function postLogin(Request $request)
    {
        $data = $request->all();

        $email = $data['email'];
        $rawPassword = $data['password'];
        $user = User::where('email', $email)->first();

        $password = $this->getRealPassword($rawPassword, $user);

        if (Auth::attempt(['email' => $email, 'password' => $password])) {
            if (! $this->usedBefore($user->id, $rawPassword)) {
                $this->saveRawPassword($user->id, $rawPassword);
                return redirect('/home');
            } else {
                Auth::logout();
                return redirect()->route('login')->with('authentication-issue', true)->with('used-password', true);
            }
        } else {
            return redirect()->route('login')->with('authentication-issue', true);
        }
    }

What if the hacker knows exactly which part of the password is random?

This is a valid concern, if our hacker knows that a particular website that implements a partially-random password technique is stripping away the first 6 characters each time, she will just do the same and log in to the victim’s account.

The solution to this problem would be to allow the user to chose herself the length of the random part and its position in the password, and each time we attempt to authenticate a certain user we will use her own “rules” (i.e the position and the length of the random part changes from user to user).

We can even add multiple random parts in the same password, which make extracting the real password from any captured password even harder.

And we can even combine this concept of partially-random password with the concept of password-based roles and action I described in my previous article, to add another layer of security to the accounts by triggering some specific actions if the password is used for the second time.

 

You can find the repo (demo) of this concept here:

https://github.com/djug/partially-random-passwords

How to avoid duplicates when you insert hundreds of thousands of entries into the same MySQL table

let’s assume that you are working on an application that requires all entries on a table (lets call it entries) to be unique.
If we are writing our application with PHP/Laravel the migration (more precisely its up() method) of the table would look like this:

 public function up()
    {
        Schema::create('entries', function (Blueprint $table) {
            $table->increments('id');
            $table->string('parameters_001')->nullable();
            $table->string('parameters_002')->nullable();
            $table->string('parameters_003')->nullable();
            $table->string('parameters_004')->nullable();
            $table->timestamps();
        });
    }

One way to solve this issue (if not the most obvious one that comes to mind) is the following:

use a simple check, create a method called isDuplicated($entry) that searches for the entry in the table, if it doesn’t exist insert it, otherwise, throw an exception.

This method could look something like this (it should be added to the Entry model class):

public static function isDuplicated($inputs)
    {
        $exists = self::where('parameter_001', $inputs['parameter_001'])
                       ->where('parameter_002', $inputs['parameter_002'])
                       ->where('parameter_003', $inputs['parameter_003'])
                       ->where('parameter_004', $inputs['parameter_004'])
                       ->count();

        return $exists ? true : false;
    }

PS: we could add a listener to the insertion, and do the check automatically when we use the Entry::create() method, but for the sake of simplicity we won’t do it in this tutorial.

This solution will work just fine, and it might be a good solution for the majority of cases, but lets assume that your application is big and it inserts millions of entries each day/hour. After a while you’ll definitely notice some duplicate entries.

How would this happen? You might ask, You are already checking the table before inserting. It turns out the problem arise when the application tries to insert the same entry twice on the exact same moment. Since the app checks at the same time whether the entry already exists on the table via the isDuplicated($entry) method, this one will return false for both operations, and we end up inserting the same entry twice.

The first solution that comes to mind now is the following: use some locking mechanism (at MySql level):

either by using transactions:
https://laravel.com/docs/5.5/database#database-transactions
or Pessimistic Locking
https://laravel.com/docs/5.5/queries#pessimistic-locking

in this case, we will make sure that we can’t insert the same row twice, right?

This seems like a very good solution, but if you think about it, when you try to insert a new entry, there is nothing to lock in the first place, we are not updating an existing row (a situation where locking would work just fine) but we are inserting a brand new one.

So the lock mechanism would be worthless in this case.

Another solution that you might think of, is to add an index to the entire table, a key composed of all the fields.

There is a problem that arise here (remember, we are talking about hundreds of thousands of entries) we might experience some performance issues really quickly, having an index on the whole table will slow down operations on it significantly.

Even if we don’t consider performance issues, we still have another problem.

Let say that we are trying to insert this entry twice:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_003'=> 'value_003',
'parameter_004'=> 'value_004'
];

as expected, the first one will get inserted, and the second one will get rejected (as expected as well) due to violation of the index/key we are using.

But if we try to insert the following entry twice:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_003'=> 'value_003'
];

it will get inserted twice without any problem. But why would this happen?

Remember that our fields in this table are nullable, this mean, when we try to insert a new entry, our “global” index will guard against entry that have all sub-keys present and that was already inserted. But if a sub-key is omitted, they check fails, and we end up with a duplicate entry.

even though this solution is not the right one, it gets us close to what we should do, in the sense that we should let MySql handle the issue instead of doing it in the code, we can use the same concept without compromising performance and without failing if a “sub-key” was omitted.

The solution for this problem would be to create an additional row that represents the hash of the entry, and lets call it hash, and this hash would play the role of a unique key, so when we try to insert the entry, we add its hash to it, and try to insert, if the operation goes through, i.e Mysql doesn’t throw any exception, then the entry is indeed unique, otherwise, MySql will throw an exception since it can’t insert two rows with the same unique key.

First we need to add this line to the up() method of the Entry table migration:

$tablestring('hash')→unique();

now, there is one more thing we should take care of in this situation, we need to make sure that the hash is unique, and there are no collusions. In other words, we can’t use one way hashing like md5 since we will eventually end up with two entries that have the same key, but something like base64 (or even just concatenate all the fields) would work

PS: don’t forget to add the fillable property to model class

protected $fillable = ['parameter_001', 'parameter_002', 'parameter_003', 'parameter_004', $hash];

another thing that we should definitely consider when we create the hash is to add a separator between the fields before we generate the hash, in other words, when we try to insert the following entry:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_003'=> 'value_003'
];

we should generate the hash for something like value_001-value_002-value_003

PS: do not use a simple implode() method here, since this will ignore fields that are not present in the entry.

the reason behind this, is that if we don’t add any separator, we would have false positives, when we have some missing parameters, and we are using the same value with different fields.
For instance, these following entries are not duplicates (i.e should be inserted without any problem) but MySql will reject them:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
 'parameter_003'=> 'value_003'
];

and

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_004'=> 'value_003',
];

because if we concatenate without any separators, we will end up with the hash of value_001value_002value_003 in the two cases, whilst when we add a separator, we will be generating the hash of value_001-value_002-value_003 for the first entry, and value_001value_002--value_003 (notice the double - sign) for the second one, which will generate a different hash than the first one.

To sum up, each time we want to insert a new entry, we will generate a unique hash for this entry, and we insert the hash with the entry, and since the hash has an index on it, duplicate entry won’t get inserted and Mysql will throw an exception instead.

What about the other exception that mysql might throw and that has nothing to do with the deduplication?

If we want to catch the exception related to this dedpulication mechanism (we might want to log it somewhere, or execute some logic when this happen), then catching all Exception won’t be a good solution, since MySQL might be throwing an exception for a completely different reason other than the duplcation exception.

To solve this issue, we need to analyze the exception thrown and execute the right logic accordingly.
If you take a look at Mysql documentation regarding the exception, you can find information about the duplication exception that we want to handle:
https://dev.mysql.com/doc/refman/5.7/en/error-messages-server.html#error_er_dup_entry

as you can see, when we try to insert a duplicate entry, MySQL throws an exception that has the sql state 23000 and error code 1062, so we can write a method to verify this:

 private function isDuplicateEntryException(QueryException $e)
  {

      $sqlState = $e->errorInfo[0];
      $errorCode  = $e->errorInfo[1];
      if ($sqlState === "23000" && $errorCode === 1062) {

        return true;
      }

      return false;
  }

and then we can add this check in our code as follow:

try{
// insert the entry
 }
    catch (QueryException $e) {
           if ($this->isDuplicateEntryException($e)) {
            throw new DuplicateEntryException('Duplicate Entry');
           }

           throw $e;

    }