The Laravel Testing 101 ebook is launched and ready!

I am extremely excited to announce the availability of the Laravel testing 101 ebook!

I have spent countless hours working on this ebook over the last months and I hope that it will help you learn how to add tests to your Laravel CRUD applications.

You can get your copy for just 19.99$ (limited time only):

https://leanpub.com/laraveltesting101

 

you can also sign up here to get 3 free chapters:

https://laraveltesting101.com/

Please let me know any feedback by tweeting at me (@djug) or simply emailing me (my first name at gmail).

How to set up your local projects to contribute to Laravel (or any PHP library) : a practical guide

Let’s imagine this scenario: You are working on a Laravel application, and you found a bug either in the framework itself or in one of the third party libraries you are using. You edit the code inside your project, or maybe you installed a fresh copy of Laravel and library that has the issue, and you edited the code there. You saved the changes, but when you went to your terminal to commit the changes, GIT just ignored what you did. You start wondering “What’s happening here?”.

 

The first solution that might pop up in your head could be to read the “How to contribute” of the framework or the library documentation, and in most cases, you’d find there the necessary steps to contribute to the framework, but I always felt that something was missing in the documentation. Let me explain what I mean by that:

Let’s take the example of Laravel here, if you decide to contribute to the framework, you’d need to clone the laravel/framework repository, push your changes to your own copy of the repository (on github), and create a PR, but what’s missing here is How would you test your changes (bug fix, new feature, …) in a local project, since if you introduce the changes in that project you just can’t commit them and create a PR for them easily, and if you change the code in the laravel/framework repository you just cloned you might end up doing a lot of copy/pasting between this repository and your project in order to test those changes.

Luckily there is a cleaner way to deal with all this. I’m not going to walk you through the usual steps of creating a PR, I assume since you are reading this article you are already familiar with that. I’ll focus on how you should set up your local environment to streamline this whole process and show you how you could fix bugs and develop new features in the framework/library and test them locally before you create the PR you are about to send.

Prepare your test project and your repository

First, you’d need to fork the laravel/framework repository and then clone it locally (the usual steps when it comes to contributing to any open source project) and then create a test application where you are going to test those changes (I’m calling it playground)

➜ laravel pwd
/home/djug/laravel
➜ laravel ls
framework playground

Make Composer load the changes from the local repository

Now, with the most important part. We need to tell the playground application to take into consideration the changes we introduce in the /home/djug/laravel/framework repository

We can do this via composer, and according to its documentation and I’m quoting here:

Loading a package from a VCS repository

There are a few use cases for this. The most common one is maintaining
your own fork of a third party library. If you are using a certain
library for your project and you decide to change something in the
library, you will want your project to use the patched version. If the
library is on GitHub (this is the case most of the time), you can
simply fork it there and push your changes to your fork. After that,
you update the project’s composer.json. All you have to do is add your
fork as a repository and update the version constraint to point to
your custom branch.

So the first part of the composer.json file (on my playground project) should look like this:

{
    "name": "laravel/laravel",
    "type": "project",
    "description": "The Laravel Framework.",
    "keywords": [
    "framework",
    "laravel"
    ],

    "license": "MIT",
    "repositories": [
        {
            "type": "vcs",
            "url": "/home/djug/laravel/framework/"
        }
    ],
    "require": {
        "php": "^7.1.3",
        "fideloper/proxy": "^4.0",
        "laravel/framework": "master",
        "laravel/tinker": "^1.0"
    },
...
}

Now each time I change something in the /home/djug/laravel/framework/ repository and commit the change (you need to commit it) and if I execute composer update on my playground project it will get the changes.

3/ Push the changes to your remote repository and create a PR

Now that your playground repository can pull the changes from your framework repository, you could update add all the changes you want to your code and then test them on a “real” project.

After you finish testing, all that you need to do is to push the changes to your remote repository (your own “copy” of the framework repository on GitHub for instance) and then create the pull request.

 

Photo by John Schnobrich on Unsplash

[Laravel Testing 101] Writing tests for guest user functionalities on a Laravel CRUD application

This is an excerpt of my upcoming ebook Laravel Testing 101. If you haven’t read the previous chapters yet (available for free here: Adding Tests to your Laravel CRUD Application: Where to Start? and here: What should we be testing in a [laravel] CRUD application?), please do so before reading this one.

Now that we have a better idea about the functionalities we should be testing in our Laravel application, let’s start with testing what a guest can do, since it is less complicated than what a logged in user can do.

As we discussed I the previous chapter, here are the functionalities related to guests that we have in the application:

  1. A guest could see all the articles when visiting /articles
  2. A guest could see a single article
  3. A guest could see a user profile
  4. A guest could not write a new article and get redirected to the signup page instead
  5. A guest could visit and get the signup page
  6. A guest could visit and get the login page

Make sure that PHPUnit is working properly with your application

Before we start writing any tests, let’s make sure that PHPUnit is working properly with your application.

The binary of PHPUnit is included in vendor/bin/phpunit in your project, so all you need to do is to execute it (from your project directory).

You should see this result:

Even though we haven’t written any tests yet, Laravel includes the following example tests:

  • /tests/Feature/ExampleTest.php
  • /tests/Unit/ExampleTest.php

P.S.: I recommend adding an alias for the command above, so you would not need to type vendor/bin/phpunit each time you want to run the tests

For example, I’m using this alias:

alias lphpunit="vendor/bin/phpunit"

1/ A guest could see all the articles when visiting /articles

Since we are going to test functionalities related to the ArticleController, let’s first create a class dedicated to this controller.

php artisan make:test ArticleControllerTest

Note that I didn’t pass the --unit flag to the command, which mean we are not creating a unit test but rather a feature test. The newly created class should be located in /tests/Feature/ArticleControllerTest.php

You can get rid of the testExample that was included with the ArticleControllerTest.

Let’s create our first tests.

When executing PHPUnit, it will look for all the public method that either start with test or that have @test in their dockblock.

So you could either use this format:

public function testGuestCouldSeeListOfArticles()
{
...
}

or this one:

/**
* @test
*/
public function it_allows_anyone_to_see_list_all_articles()
{
...
}

I prefer the second one since it is much easier to read.

I’ll start with the most basic test. I just want to make sure whenever I hit the /articles route, I get a valid page back.

/**
* @test
*/
public function it_allows_anyone_to_see_list_all_articles()
{
	$response = $this->get(route('get_all_articles'));
	$response->assertSuccessful();
}

Save the file and run PHPUnit (either using vendor/bin/phpunit or with the lphpunit alias we created earlier).

Our tests passed 🙂

Note that, even though we wrote just one test and one assertion PHPUnit tells us that we have 3 tests and 3 assertions.

Note: An assertion is testing one single “thing”, a test could
contain multiple assertions
. The test we wrote above contains just a
single assertion $response->assertSuccessful();

The reason behind this, is that PHPUnit will run all the tests in the /tests directory. Running all the tests each time is not a problem at the stage we are in right now, but if you want to run just a single test, you can pass the name of the test (the name of the method) as a parameter to the --filter flag like this:
lphpunit --filter=it_allows_anyone_to_see_list_all_articles

… and yes, as you might have guessed it you could add an alias for this command in order to save time next time you want to run just a single test.

alias lphpunit="vendor/bin/phpunit"  
alias lphpunitf="lphpunit --filter="

lphpunitf it_allows_anyone_to_see_list_all_articles

As you can tell, we want to test more than just getting a valid page, we want to make sure that we are getting the right page.

We can test this with the following steps:

  • make sure that we are getting the right view
  • make sure that the view contains the variables needed for this page

Laravel provides two method to test the above:

$response->assertViewIs('articles.index');
$response->assertViewHas('articles');

Our test class should now look like this:

<?php
namespace Tests\Feature;

use Tests\TestCase;
use Illuminate\Foundation\Testing\WithFaker;
use Illuminate\Foundation\Testing\RefreshDatabase;

class ArticleControllerTest extends TestCase
{
	/**
	* @test
	*/
	public function it_allows_anyone_to_see_list_all_article()
	{
		$response = $this->get(route('get_all_articles'));

		$response->assertSuccessful();
		$response->assertViewIs('articles.index');
		$response->assertViewHas('articles');
	}

}

Now that we can test that we are getting the right view (with the right variable), we no longer need to keep the first assertion, since it is implicit.

2/ A guest could see a single article

Now that we tested that a guest user could see the list of all articles, let’s make sure that she could view individual articles as well.

In order to ensure that this functionality (showing individual articles to guest users) works as expected, we would need the following steps:

  1. Get an article to view (randomly)
  2. Generate the route to this article and send a GET request to it
  3. Make sure that we are getting the right view (in this case articles.view)
  4. Make sure that the view returned contains a variable named $article
  5. Make sure that we are getting the article we wanted to access and not another one.

Our test should look like this:

/**
* @test
*/
public function it_allows_anyone_to_see_individual_articles()
{
	$article = Article::get()->random();
	$response = $this->get(route('view_article', ['id' => $article->id]));
	
	$response->assertViewIs('articles.view');
	$response->assertViewHas('article');
	$returnedArticle = $response->original->article;
	$this->assertEquals($article->id, $returnedArticle->id, "The returned article is different from the one we requested");
}

Note:
We can access the returned view thought the $response->original variable

You might ask why we are doing all these steps for a such simple feature. The feature is indeed simple, and its tests are simple as well… simple, but not trivial.

We are doing all these steps to insure the following:

  • We want to test accessing a random article each time (we would not want to always request the same ID or the same article), because we might have an issue in the code that makes the application always return the same article. Imagine for instance, that instead of searching for a specific article, for some reason we updated our code to use Article::first(), we wouldn’t be able to detect this issue if we keep returning the same article (using the same ID) over and over again.
  • It is highly recommended to use routes in your tests instead of URLs, because if you ever change the structure of your URL, you wouldn’t need to update your tests.
  • We also want to make sure that we are indeed getting the article we are requesting, because we might get the right view that includes the variable we are looking for, but it might contain a different article from the one we requested.

3/ A guest could see a user profile

This one should look a lot like the previous test since the concept is the same (access a model and return it), but we are accessing a user instead of an article.
Since we are not testing articles here, we should create a new test class first:

php artisan make:test UserControllerTest

Then all what we need to do is to add the following test:

/**
 * @test
 */
public function it_allows_anyone_to_see_users_profiles()
{
    $user = User::get()->random();

    $response = $this->get(route('show_user_profile', ['id' => $user->id]));

    $response->assertViewIs('users.show');
    $response->assertViewHas('user');

    $returnedUser = $response->original->user;

    $this->assertEquals($user->id, $returnedUser->id, "The returned user is different from the one we requested");
}

4/ A guest could not write a new article and gets redirected to the signup page instead

This one (and the remaining ones in this chapter) are much simpler, since they do not require accessing the DB.

In order to test this functionality, we need the following steps:

  • attempt to access the create_new_article route
  • test whether we got redirected to the login page [login or signup page ?]

The test should look like this:

/**
* @test
*/
public function it_prevent_non_logged_in_users_from_creating_new_articles()
{
	$response = $this->get(route('create_new_article'));
	$response->assertRedirect('login');
}

5. A guest could visit and get the signup page and 6. A guest could visit and get the login page

These two tests are even simpler to write, since we are just checking that when we attempt to visit the login and signup page, we get valid pages. Since we are using the Laravel built-in authentication controller, we would not need to test the authentication ourselves.

We would need a new test class for these two tests as well. We could either create a dedicated class just for them. Usually I put all the “page tests” (i.e tests that ensure we are getting valid pages when we hit certain URL) in a PagesControllerTest (especially if I have a controller named PagesController) ; or just create a test class for HomeController since in most cases I add the logic of the pages I tests to this class.

The two test cases should look like this:

/**
* @test
*/
public function it_returns_register_page()
{
	$response = $this->get(route('register'));
	$response->assertSuccessful();
}
/**
* @test
*/
public function it_returns_login_page()
{
	$response = $this->get(route('login'));
	$response->assertSuccessful();
}

Also, instead of just checking whether we are getting a valid page (which is more than enough in this situation), we also check that we are getting the right views like this:

/**
* @test
*/
public function it_returns_register_page()
{
	$response = $this->get(route('register'));
	$response->assertViewIs('auth.register');
}
/**
* @test
*/
public function it_returns_login_page()
{
	$response = $this->get(route('login'));
	$response->assertViewIs('auth.login');
}

How would tests detect breaking changes in the code base?

As we discussed in a previous chapter, one goal for writing tests is to ensure that the functionalities of the application will keep working the way we intended when we first built them.

I’d like to show just one quick example of how the tests will inform us that we are introducing a new code that is changing the behavior of the application (a breaking change).

Let’s assume that after a few weeks of working on this application we decided for some reason to update the constructor of the ArticleController from this:

public function __construct()
{
	$this->middleware("can:manage,article")->only('edit', 'update', 'delete');
	$this->middleware("auth")->only('create');
}

to this:

public function __construct()
{
$this->middleware("can:manage,article")->only('edit', 'update', 'delete');
$this->middleware("auth");
}

the only change is deleting ->only('create') from the second line of the constructor.

This might either happen by accident (a teammate didn’t see the value of protecting just one action with this middleware) or it was done intentionally to prevent guest users from reading articles before signing in.

If we run the tests, we would get this:

Before writing any tests of the application, chances are that you wouldn’t notice the breaking change before a while. Maybe the breaking change will even get deployed without anyone noticing it, since you’d be using your application as a logged in user most of the time, and you wouldn’t think that you’d need to test the guest functionalities after every small change to the application.

But with the tests, any breaking change will be detected right away without even needing to test the application manually. And if you have a CI (Continuous Integration) set up with your project (we will be exploring how to set it up later in this ebook), you wouldn’t even be able to merge/deploy without fixing the issue first.

Conclusion

In this chapter, we explored the different steps needed to test guest users functionalities. We’ve seen that even though the tests are simple, they sometimes require extra steps to ensure that we are testing the right thing, and we are not missing some edge cases (especially when we introduce a change that might break the application). We’ve also seen how tests would detect breaking changes in the code base.

In the upcoming chapters, we will be exploring tests related to logged in users which could be a little more challenging than the tests we’ve seen so far.

If you want to follow along, and get notified of any progress with this ebook (new free chapters for instance), please sign up here: https://laraveltesting101.com/

What should we be testing in a [laravel] CRUD application?

This is an excerpt of my upcoming ebook Laravel Testing 101. If you haven’t read the previous chapter yet (available for free here: Adding Tests to your Laravel CRUD Application: Where to Start?), please do so before reading this one.

 

Now that you understand that you should be looking at tests from a different angle, and that we should test mainly controllers, you might start to ask, “… but what should I be testing exactly?”

In this chapter, we are going to answer this question and describe what we will be testing in the rest of the ebook.

But first, let’s get the demo application the book is built upon up and running.

Clone the following repo

https://github.com/djug/laraveltesting101

and run the application after you follow all the steps in the readme file.

After you fire up your server/application, you should get this front page:

So far, here is what you can do within the application:

  • click on either “login” or “register” to see the login and signup pages.
  • click on “all articles” and you’ll get to /articles, that contains all the articles generated by the seeder.

  • click on “write a new article” (/new-article) and you’ll get redirected to the signup page (since adding a new article requires signing up).
  • from the “/articles” page, you could either click on articles name (example /articles/15), or user names (/users/5) to view the articles or their authors profiles.

After signing up you’d be able to view the /new-article route (which was redirecting to the sign up page when we tried it earlier without signing in)

After saving your article, you can view it, and since it is your own article, you would see the “edit article” button at the end of the page (you won’t see it on articles that you do not own).

When clicking on this button, you get the edit form of the article, where you can edit and save the article or you could delete it.

When updating the article you’ll get redirected to it again, with a success message.

But if you delete it, you’ll get redirected to the “all articles” page with a different success message.

You won’t see an “edit” button if you visit articles that you didn’t create yourself, but if you try to access the edit page of an article you don’t own via the URL directly (ex: articles/16/edit), you’ll get a 403 page.

Let’s go back to the “Write a new article” page and try to add a new article with a short title and/or a short body (shorter than 10 characters).

You’ll get an error message. And the same thing happens when editing.

So even though the application is small, there are multiple things we need to test to ensure that the application is working as expected.

And here is the list:

  1. A guest could visit and get the signup page
  2. A guest could visit and get the login page
  3. A guest could see all the articles when visiting /articles
  4. A guest could see a user profile
  5. A guest could not write a new article and get redirected to the signup page instead
  6. A user (i.e. signed up user) could write an article
  7. A user could access the edit page of her own articles
  8. A user could save the changes of her own articles and get redirected to the article page after editing with a success message
  9. A user could delete her own articles and get redirected to the all articles page with a success message
  10. A user should not see the edit button on the article she doesn’t own
  11. A user could not edit articles she doesn’t own
  12. A user could not delete articles she doesn’t own
  13. When creating an article, the article cannot have short title or body
  14. When updating an article, the article cannot have short title or body

So if we manage to automate testing the above list, we could be assured that each time we add a new feature or change a code, the application will keep working as expected.

In the next chapter, we will start writing tests with functionalities related to guests users.

If you want to follow along, and get notified of any progress with this ebook (new free chapters for instance), please sign up here: https://laraveltesting101.com/

[Laravel tips] How to redirect users to a specific URL/route when the validation fails

If you are using Form Request to validate your forms before you persist them in the database, when the validation fails, the user is usually redirected back to the same page that initiated the request.

For instance if you are on /new-article page, after you submit the article you are creating, the user will be redirected back automatically to this very same URL if the data she entered failed to pass the validation. This is the desired behavior in majority of the cases.

Lately, I was working on a form where I needed to redirect the user back to another route.

I was working on an application where we have different settings the user could update (from the settings page), and since it wasn’t desirable to send all the data in this page at once, validate it and then update the user settings, I created multiple “small form” each one is in a section of an accordion that updates just a group of related settings. So I could for instance visit /settings to view the settings page with all the sections/cards collapsed, or I could visit (/settings/personal-info) to open the settings page on that particular section.

So I might be visiting /settings/personal-info and then I’d scroll down to another section to update it (privacy for instance). In this case, if the validation fails, the user will be redirected to /settings/personal-info instead of /settings/privacy.

PS: I know this might not be the best way to handle this kind of the situation, but bear with me here, the goal is to explain how we could redirect the user to a different route 🙂

Luckily, we can redirect the user to the desired URL/route by setting up some propriety in the form request class.

Take a look at /vendor/laravel/framework/src/Illuminate/Foundation/Http/FormRequest.php

and you could find these 3 properties:

/**
* The URI to redirect to if validation fails.
*
* @var string
*/
protected $redirect;

/**
* The route to redirect to if validation fails.
*
* @var string
*/
protected $redirectRoute;
  
/**
* The controller action to redirect to if validation fails.
*
* @var string
*/
protected $redirectAction;

So all we need to do is to give the value we want to one (just one) of this properties.

protected $redirectRoute = 'get_all_articles';

PS: if you want to know what would happen if you set up more than one, take a look at the protected function getRedirectUrl() method on the same class.

One issue you might face in this situation is when you want to pass a value to redirectRoute, but you want to pass a route with a parameter like

Route::get('articles/{article}', '[email protected]')→name('view_article');

The solution is to set up the value of this property in the constructor like this:

public function __construct()

{
	$id = ….;
	$this->redirect = route('view_article', $id);
}

Note that I’m using $redirect here instead of the redirectRoute.

I hope you’ll find this tip useful 🙂

Adding Tests to your Laravel CRUD Application: Where to Start?

This is an excerpt from my upcoming ebook Laravel testing 101

Imagine you are working on a Laravel side-project or you just joined a team working on a Laravel application, and it currently doesn’t have a single test. You want to change that, and add tests to the application. If you find yourself in this situation, you might not even know where to start. “Should I unit test everything? Should I just test the most critical parts of the application?” You might even find yourself asking “Why do I need tests in the first place?

I am writing an ebook to answer all these questions for you, and this is the introductory chapter. My main goal with this ebook  is to try to answer these questions, and I will describe my process in writing tests for some of the Laravel applications I’m working on (mainly CRUD applications).

 

What you are going to read in this chapter and the following ones might not necessarily be the optimal way of testing or even the best practice, but rather just one possible way of doing testing in Laravel [CRUD] applications, the way I am approaching testing these days.

 

Why we are writing fewer tests than necessary

For me, testing is just like physical exercise. We all know that we should be exercising (or exercising more), but very few of us exercise on a regular basis, if at all.

Many people decide they’ll be exercising every single day as part of their new year resolution. They get a yearly subscription at their local gym to prove how serious they are, they might even hit the gym 5 days in a row the first week of the year. But after the first weekend, they start going less and less, and end up just giving up because their body aches so much and they don’t see any improvement yet (they worked out for a WHOLE week, they should at least notice a quick drop in their weights, no?). The other scenario is that they might feel overwhelmed with all the choices they find at the gym: Should I focus just on cardio? Maybe lift some weights? Or maybe I should do both? Maybe I’ll just focus on my biceps… A classic case of paralysis by analysis.

Almost the same thing happens with developers and testing.

With every new project, they decide they’ll unit test the sh*t out of it. They decide to test every single class and method (even the private ones), they would even buy some ebooks or online courses to learn how to implement a testing approach in their project.

They’ll start implementing TDD, for instance, in the first week of the project, but after the first weekend passes by, they’ll start thinking about giving up. Not only they don’t see any improvement in their development process or in the end results (they have been TDDing everything for a whole week for God’s sake), but also they’ll start writing tests for just a method or two on each class, and then start to question if this approach is the right one or if they should try something else entirely, and then, the inevitable happens, they drop everything.

In other words, testing is just like exercising, we tend to think that we should do it perfectly (exercising every single day), or we shouldn’t bother at all. When in reality, all what we need is some good practices, and a change in the way we view and approach testing. We have to understand why we need tests in the first place, and learn how we can get the most out of them without doing too much.

 

Why do we need tests in the first place?

If you ask this question to a group of developers, you might not get the same answer twice. Personally, I write tests for one simple reason: to make sure that the application will keep working in the exact same way it was intended to when each piece of it was written. In other words, each time I introduce a new change to the code base or add new feature, I want to make sure that I’m not breaking any current functionalities.

If you think about it, each time you introduce a change and you start testing your app in the browser, you are just checking that everything is still working.  The method I’m describing here will just help you automate this whole process, i.e instead of opening random pages in the app to see whether they are still working, you execute your test suite, and confirm that everything is still working as expected.

Let me repeat it one more time, since this is really important, and all what follows is built upon this idea: write tests to make sure that the current functionalities keep working as expected when we introduce changes

In other words, testing should not be about proving that the code is 100% correct, all we need is to make sure that we are not going to break it or change its behavior in the future.

So, what should I be testing exactly?

As you can probably tell from the description above, we should be testing only controllers in this phase. If you think about it, if all the controllers are behaving exactly how they are supposed to, it is most likely that all the underlying code of the application (models, repositories, services…) is working just fine, unless your test suite doesn’t cover all the use cases of the controllers.

Structure of the ebook

What I don’t like in the majority of online resource about testing, is that they present each idea/concept in isolation. When usually, especially for a beginner, it is hard to tell how all these concepts fit together.

So instead of doing this, I’ll be adding tests to a small demo application I built for the purpose of this ebook.

It is a small application that allows users to sign up, and post articles. Each user can edit and delete their own articles, and read what other users post on the website.

The application is simple, but as you’ll see in the upcoming chapters, it contains enough code/functionalities to demonstrate the all the necessary ideas you would need in testing almost any Laravel CRUD application.

If you are interested, and you want to follow along, please visit laraveltesting101.com and sign up. You’ll be notified every time I publish something new.

 

laraveltesting101.com

[Laravel tips] How to temporarily change a laravel environment variable without updating the .env file

Imagine the following scenario: you are working on a laravel application that dispatches some jobs to a queue. You notice in your log file that a specific job is throwing an exception, but you can’t debug it directly (with dd() for instance), since the job is consumed by a worker elsewhere.

One way to debug this issue would be to change the QUEUE_DRIVER environment variable in your .env file to sync, debug, and then revert back the change after you finish.

Or imagine that you have a test database and you need to run the migrations on it first, and you’d need to update the .env file to use the test database first, run the migrations and then revert back the change.

As you can see, in both cases, all what we need to do is to update the environment variable temporarily. And usually, it takes some time to update the .env file (you might even need to clear the configuration or restart some workers).

Luckily, it is possible to pass an environment variable to linux commands, and these variables will only be used for the current execution.

In the previous example, all what we have to do is to execute the command as follows:

QUEUE_DRIVER=sync php artisan my-artisan-command-here

You can test it yourself with tinker.

You can even pass multiple environment variables like this:

QUEUE_DRIVER=sync DB_DATABASE=testing php artisan my-artisan-command-here

As you can see, this is not a laravel specific trick, and you can use it whenever you find yourself in a situation where you need to change an environment variable temporarily.

I hope you’ll find this useful.

[PoC] Partially random passwords: or how to protect users passwords from keyloggers with partially random passwords

One of the issues we all face when we login to some online accounts especially on public computers, or on any computer that we do not own, is that there is always a risk to get our passwords stolen especially with keyloggers. If a hacker gets a “copy” of your password, she can log in to your account and do whatever she wants.

As always, using 2 factor authentication can mitigate this issue, since the hacker needs to access your phone as well. But what if using 2FA is not an option, and we want to protect the user even in this case, can we detect,when the hacker tries to login, that it is not the real user but rather somebody who stole her password who is trying to log in?

Partially-Random passwords

One solution for this problem is to use partially-random passwords with a random part that is different each time.
What I mean by that is the following:
let’s assume that my password is 123456789, but when I try to login I will not use this password directly, but rather I’d add some random string to the beginning or to the end (or even in the middle) like this:
564564123456789

and when we receive this password (let’s call it the “raw password”), we will strip away the random part and use the remaining part as a password, and we save a hashed version of this raw password in a used_passwords table. And next time the same user tries to login, we check if this raw password (the real password + the random part) was used before, and we deny access to the account if we find it.
The code will look like this:

public function postLogin(Request $request)
    {
        $data = $request->all();

        $email = $data['email'];
        $rawPassword = $data['password'];
        $user = User::where('email', $email)->first();

        $password = $this->getRealPassword($rawPassword, $user);

        if (Auth::attempt(['email' => $email, 'password' => $password])) {
            if (! $this->usedBefore($user->id, $rawPassword)) {
                $this->saveRawPassword($user->id, $rawPassword);
                return redirect('/home');
            } else {
                Auth::logout();
                return redirect()->route('login')->with('authentication-issue', true)->with('used-password', true);
            }
        } else {
            return redirect()->route('login')->with('authentication-issue', true);
        }
    }

What if the hacker knows exactly which part of the password is random?

This is a valid concern, if our hacker knows that a particular website that implements a partially-random password technique is stripping away the first 6 characters each time, she will just do the same and log in to the victim’s account.

The solution to this problem would be to allow the user to chose herself the length of the random part and its position in the password, and each time we attempt to authenticate a certain user we will use her own “rules” (i.e the position and the length of the random part changes from user to user).

We can even add multiple random parts in the same password, which make extracting the real password from any captured password even harder.

And we can even combine this concept of partially-random password with the concept of password-based roles and action I described in my previous article, to add another layer of security to the accounts by triggering some specific actions if the password is used for the second time.

 

You can find the repo (demo) of this concept here:

https://github.com/djug/partially-random-passwords

[PoC] Password-Based user roles and triggers/actions (or how to use different passwords for multiple roles for the same account)

We have all seen a movie or two where a villain forces some innocent people, to give him the password to access her ultra secret account (a nuclear facility, or a Swiss bank account), the villain ends up getting access to the account and transfers all the money to his own account, or launch a nuclear weapon.
We have also all heard about authorities in some countries that force their citizens or even visitors to hand them their social media accounts and passwords. Usually the “victims” don’t have any choice and they end up providing their passwords.

One way to prevent this is maybe to activate multi level authentication on your account (2FA), and leave your phone, where you receive the one time password, at home when you travel, but this might not be a real solution in too many cases.

What if instead of all this, we have systems/websites with multiple roles/password for each account, and instead of using just one password all the time, we can use another one to tell the system “Hey system, when I use this particular password it means that I’m not in a secure place (or I’m forced to hand out my password) so please show just a restricted version of my account”, or even better “Hey system, when I use this particular password just disable my account, and do not accept any request to activate it for 30 days”.

I’ll explain in this article how we would implement such a system , for the sake of simplicity, I’ll focus just on the main concepts, and I’ll be using just a single controller, a single table for the users and their roles, and even use just 3 types of account: master, restricted and a trigger. I’ll be also using PHP/Laravel, but you can implement the same idea with the language of your choice.

Add multiple roles for the user

In order to add multiple roles for the user, we will be creating additional [sub-]account to the user.
To do so, we will create a users table migration that contains:

$table->string('email')->unique()->nullable();
$table->integer('master_account_id')->unsigned()->nullable();
   $table->foreign('master_account_id')->references('id')->on('users');
$table->string('type')->nullable();

as you can can see here, the email is nullable, so we can create new accounts without an email (we can prevent the users from creating new “main” accounts without a password with a validation layer).
We are also referencing a master account, and the type of the account.

We can create a main account and two sub-accounts like the following example:

Basic/classical authentication

When we build a classical authentication in a Laravel application (without using the built it authController), we usually check if the password we get corresponds to the email like this:

public function postLogin(Request $request)
{
   $data = $request->all();
   $email = $data['email'];
   $password = $data['password'];
   if (Auth::attempt(['email' => $email, 'password' => $password]) {
      // redirect the user to the dashboard
   } else {
   // redirect the user back to the login page
   }
…
}

in our new authentication system we will change that a little bit

first, we get all the accounts that belongs the email address:

$emailOwner = User::where('email', $email)→first();
$users = User::where('master_account_id', $emailOwner->id)->get();

and then we attempt to login to them one by one with the password we received:

foreach ($users as $user) {
   if (Auth::attempt(['id' => $user->id, 'password' => $password])) {
      return redirect('/home');
   }
}
// if we finish the loop without finding a match, redirect the user back to the login page

note here that we are attempting the authentication with the user ID (not the email) and the password, and as soon as we find a match, we login with the found account.

As you can see here we created a multi-roles account and we chose the role based on the password the user used.

Add password-based triggers/actions

let’s add one more thing, lets add a way to trigger some actions when we login with one of the roles. To keep the example simple, let’s disable all the accounts associated with an email address if we authenticate with a specific password.

First lets add an additional field to the user account:

$table->boolean('disabled')->default(false);

and the PostLogin method (the one we use to authenticate the user) will become like this:

function postLogin(Request $request)
{
    $data       = $request->all();
    $email      = $data['email'];
    $emailOwner = User::where('email', $email)->first();

    if (!$email) {
        return redirect()->route('login')->with('authentication-issue', true);
    }

    $password   = $data['password'];
    $users      = User::where('master_account_id', $emailOwner->id)->get();
    foreach ($users as $user) {
        if (Auth::attempt(['id' => $user->id, 'password' => $password])) {
            if ($user->type == "trigger") {
                $this->trigger($emailOwner->id);
            }
            if ($user->disabled) {
                return redirect()->route('login')->with('account-disabled', true);
            }
            return redirect('/home');
        }
    }

    return redirect()->route('login')->with('authentication-issue', true);
}

function trigger($userId)
{
    $users = User::where('master_account_id', $userId)->get();
    foreach ($users as $user) {
        $user->disabled = true;
        $user->save();
    }
}

when we login to our account with the password associated with the trigger account, the master account and all its sub-accounts get disabled.

You can find the demo in this repo:
https://github.com/djug/password-based-roles-actions

 

How to avoid duplicates when you insert hundreds of thousands of entries into the same MySQL table

let’s assume that you are working on an application that requires all entries on a table (lets call it entries) to be unique.
If we are writing our application with PHP/Laravel the migration (more precisely its up() method) of the table would look like this:

 public function up()
    {
        Schema::create('entries', function (Blueprint $table) {
            $table->increments('id');
            $table->string('parameters_001')->nullable();
            $table->string('parameters_002')->nullable();
            $table->string('parameters_003')->nullable();
            $table->string('parameters_004')->nullable();
            $table->timestamps();
        });
    }

One way to solve this issue (if not the most obvious one that comes to mind) is the following:

use a simple check, create a method called isDuplicated($entry) that searches for the entry in the table, if it doesn’t exist insert it, otherwise, throw an exception.

This method could look something like this (it should be added to the Entry model class):

public static function isDuplicated($inputs)
    {
        $exists = self::where('parameter_001', $inputs['parameter_001'])
                       ->where('parameter_002', $inputs['parameter_002'])
                       ->where('parameter_003', $inputs['parameter_003'])
                       ->where('parameter_004', $inputs['parameter_004'])
                       ->count();

        return $exists ? true : false;
    }

PS: we could add a listener to the insertion, and do the check automatically when we use the Entry::create() method, but for the sake of simplicity we won’t do it in this tutorial.

This solution will work just fine, and it might be a good solution for the majority of cases, but lets assume that your application is big and it inserts millions of entries each day/hour. After a while you’ll definitely notice some duplicate entries.

How would this happen? You might ask, You are already checking the table before inserting. It turns out the problem arise when the application tries to insert the same entry twice on the exact same moment. Since the app checks at the same time whether the entry already exists on the table via the isDuplicated($entry) method, this one will return false for both operations, and we end up inserting the same entry twice.

The first solution that comes to mind now is the following: use some locking mechanism (at MySql level):

either by using transactions:
https://laravel.com/docs/5.5/database#database-transactions
or Pessimistic Locking
https://laravel.com/docs/5.5/queries#pessimistic-locking

in this case, we will make sure that we can’t insert the same row twice, right?

This seems like a very good solution, but if you think about it, when you try to insert a new entry, there is nothing to lock in the first place, we are not updating an existing row (a situation where locking would work just fine) but we are inserting a brand new one.

So the lock mechanism would be worthless in this case.

Another solution that you might think of, is to add an index to the entire table, a key composed of all the fields.

There is a problem that arise here (remember, we are talking about hundreds of thousands of entries) we might experience some performance issues really quickly, having an index on the whole table will slow down operations on it significantly.

Even if we don’t consider performance issues, we still have another problem.

Let say that we are trying to insert this entry twice:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_003'=> 'value_003',
'parameter_004'=> 'value_004'
];

as expected, the first one will get inserted, and the second one will get rejected (as expected as well) due to violation of the index/key we are using.

But if we try to insert the following entry twice:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_003'=> 'value_003'
];

it will get inserted twice without any problem. But why would this happen?

Remember that our fields in this table are nullable, this mean, when we try to insert a new entry, our “global” index will guard against entry that have all sub-keys present and that was already inserted. But if a sub-key is omitted, they check fails, and we end up with a duplicate entry.

even though this solution is not the right one, it gets us close to what we should do, in the sense that we should let MySql handle the issue instead of doing it in the code, we can use the same concept without compromising performance and without failing if a “sub-key” was omitted.

The solution for this problem would be to create an additional row that represents the hash of the entry, and lets call it hash, and this hash would play the role of a unique key, so when we try to insert the entry, we add its hash to it, and try to insert, if the operation goes through, i.e Mysql doesn’t throw any exception, then the entry is indeed unique, otherwise, MySql will throw an exception since it can’t insert two rows with the same unique key.

First we need to add this line to the up() method of the Entry table migration:

$tablestring('hash')→unique();

now, there is one more thing we should take care of in this situation, we need to make sure that the hash is unique, and there are no collusions. In other words, we can’t use one way hashing like md5 since we will eventually end up with two entries that have the same key, but something like base64 (or even just concatenate all the fields) would work

PS: don’t forget to add the fillable property to model class

protected $fillable = ['parameter_001', 'parameter_002', 'parameter_003', 'parameter_004', $hash];

another thing that we should definitely consider when we create the hash is to add a separator between the fields before we generate the hash, in other words, when we try to insert the following entry:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_003'=> 'value_003'
];

we should generate the hash for something like value_001-value_002-value_003

PS: do not use a simple implode() method here, since this will ignore fields that are not present in the entry.

the reason behind this, is that if we don’t add any separator, we would have false positives, when we have some missing parameters, and we are using the same value with different fields.
For instance, these following entries are not duplicates (i.e should be inserted without any problem) but MySql will reject them:

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
 'parameter_003'=> 'value_003'
];

and

['parameter_001'=> 'value_001',
'parameter_002'=> 'value_002',
'parameter_004'=> 'value_003',
];

because if we concatenate without any separators, we will end up with the hash of value_001value_002value_003 in the two cases, whilst when we add a separator, we will be generating the hash of value_001-value_002-value_003 for the first entry, and value_001value_002--value_003 (notice the double - sign) for the second one, which will generate a different hash than the first one.

To sum up, each time we want to insert a new entry, we will generate a unique hash for this entry, and we insert the hash with the entry, and since the hash has an index on it, duplicate entry won’t get inserted and Mysql will throw an exception instead.

What about the other exception that mysql might throw and that has nothing to do with the deduplication?

If we want to catch the exception related to this dedpulication mechanism (we might want to log it somewhere, or execute some logic when this happen), then catching all Exception won’t be a good solution, since MySQL might be throwing an exception for a completely different reason other than the duplcation exception.

To solve this issue, we need to analyze the exception thrown and execute the right logic accordingly.
If you take a look at Mysql documentation regarding the exception, you can find information about the duplication exception that we want to handle:
https://dev.mysql.com/doc/refman/5.7/en/error-messages-server.html#error_er_dup_entry

as you can see, when we try to insert a duplicate entry, MySQL throws an exception that has the sql state 23000 and error code 1062, so we can write a method to verify this:

 private function isDuplicateEntryException(QueryException $e)
  {

      $sqlState = $e->errorInfo[0];
      $errorCode  = $e->errorInfo[1];
      if ($sqlState === "23000" && $errorCode === 1062) {

        return true;
      }

      return false;
  }

and then we can add this check in our code as follow:

try{
// insert the entry
 }
    catch (QueryException $e) {
           if ($this->isDuplicateEntryException($e)) {
            throw new DuplicateEntryException('Duplicate Entry');
           }

           throw $e;

    }