Running Multiple CLI Commands in One Terminal for Laravel Development

While working on Let Them Eat Cake 🍰 , I noticed when I was end-to-end testing the app locally, I need a few different things running at once:

  • Laravel Queue Worker – running queued jobs
  • Expose – Proxy for allowing external webhooks needed for my app
  • Stripe CLI – Passing Stripe webhooks to my app without using a proxy
  • Laravel Mix – Bundling static assets

Running each of these commands separately isn’t that big of a deal, but it can be a hassle to remember to start them all. This was very apparent yesterday as my computer decided it wanted to reboot every time it went to sleep! A short trip around town running errands with my wife and my computer restarted 3 times in a couple of hours; frustrating!

Today I decided to look into some potential solutions. I had a few basic requirements while I was looking around.

  1. It needs to be easy to start and modify for each new project – this also implies some flexibility. No sudo password required nonsense.
  2. I can stop all running processes with SIGINT (CTRL+C)
  3. All process output is still visible so I can see compile errors or bad requests.
  4. Process output is in one terminal window/tab (optional – but I wanted it!)

A little searching and some testing later and I think I’ve come to a nice solution that gives me plenty of control and flexibility.

One of the first tools I discovered was moreutils, a package you can install with brew that essentially adds a handful of helpful CLI utilities. The name is play on coreutils, a set of packages included in most Linux distros. Specifically, there is a package called parallels that seemed to be just what I was looking for. It has lots of extra features and can start lots of parallel processes and close them all at once. The reason I ultimately chose not to go this route, was in case I work on any of my projects with other developers. Given the simplicity of my use case compared to the feature set of the package, it was honestly overkill. This also means my project is that much more portable between machines and developers. I keep my own dotfiles, but if I can avoid complexity, I like to try that route first. 🙂

The Solution

Ultimately, I settled on a little shell script courtesy of this Ask Ubuntu thread. Here is my adaptation:

#!/bin/bash
sh ./startexpose &
P1=$!
php artisan queue:listen &
P2=$!
stripe listen --forward-to eat-cake.test/stripe/webhook &
P3=$!
wait $P1 $P2 $P3

A quick rundown. The first process is essentially another wrapper with some Expose configuration details. The output from this is a big tail-ing table of incoming requests. The second is my queue worker. I opt for queue:listen here so the worker will automatically restart itself when any code in my application changes. Output from this command is a line for every queued job that is processed. Third is the Stripe CLI, which needs to be configured beforehand, but is a once every 90 day thing. Seeing as it has access to my payment data, I’m okay with the hassle tradeoff.

I’m still debating on whether or not to include my Laravel Mix watch command in this, and I might add a runtime flag for it, but without trying it for any reasonable length of time yet, I’m guessing I’ll need to start/stop the watcher to do production compiles more often than anything else. The other commands can run mostly unmonitored unless I need to check something, but I tend to get a lot more feedback when I inevitably screw up the closing brackets in a JS file. 😬

Hope this helps you out! Feel free to ask any questions here or send me a tweet @DaronSpence. ✌️

Migrating Data and Merging Models in Laravel

On one of my side projects, Let Them Eat 🍰 , I recently needed to do some migrations to combine / merge two models. I originally had optimized a bit too much, and later realized things would be a lot simpler if I only had one model.

There wasn’t a ton of info about this online, so I’m going to do my best to try and explain what I did. For this migration, I was using Laravel 8.

So turns out, merging models is kind of a pain! In my case, I was merging a SlackUser::class model with the default User::class model that ships with Laravel. From a data perspective, it wasn’t too bad. I needed to add a few columns to the User table that were previously on the SlackUser table. The issues arose when I realized there were a lot of places in my code that were reliant on accessing each model off of a relationship of the other. So lots of calls to $user->slackUser and $slackUser->user intermingled across the app, dependent on what I was doing at the time.

Since my ultimate goal was to completely delete any reference to a SlackUser, I had to take a careful approach when modifying the database.

First, I added the extra columns I needed to the users table.

Schema::table('users', function (Blueprint $table) {
  $table->string('slack_user_id')->after('id')->nullable();
  $table->foreignId('team_id')->after('slack_user_id')->nullable()->constrained()->onDelete('cascade');
  $table->boolean('is_owner')->after('team_id')->default(false);
  $table->string('avatar_url')->after('email')->nullable();
  $table->string('timezone')->after('avatar_url')->nullable();
  $table->boolean('is_onboarded')->after('timezone')->default(false);
  $table->softDeletes();
});

I continued in much the same process for other tables that needed to be modified.

After the tables were modified, I queried all of the SlackUsers and looped over them to create new User models if they didn’t have one already. In my app, a User model was only created if the person logged into the webapp, otherwise they would happily live on as only a SlackUser. Now, everyone gets a user model, and I don’t really care if they log in or not!

Some advice on the Shifty Coders Slack recommended not to rely on Eloquent here. This makes sense as in a future release, I’ll be completely removing the SlackUser model, so if I relied on Eloquent for the migration, it would throw an error if I ever deleted that class. Here’s what the migration looked like:

DB::table('slack_users')->get()->map(
	function ($slackUser) {
		$userId = $slackUser->user_id;
		$slackUser = Arr::except((array) $slackUser, ['id', 'created_at', 'updated_at', 'user_id']);
		$user = User::findOrNew($userId);
		if (!$user->exists) {
			$user->name = $slackUser['slack_user_id'];
			$user->password = Hash::make(random_bytes(20));
		}
		$user->fill($slackUser);
		$user->save();
	}
);

This is all pretty straightforward. For each SlackUser, check if they have a User model already, and if not, create a new User and give them a random password. My app doesn’t actually use passwords for authentication, instead relying solely on Slack Oauth, so the password field is irrelevant. In the future I may want to allow for other auth methods, so I left it for the sake of simplicity.

After all of the users were migrated, I could then go about the business of updating other models that used the SlackUser as a foreign key. In my case, I had messed up in the original migrations and not enforced those foreign keys, but if they are enforced in your app, you’ll need to drop the foreign key before you go about migrating all of this data around.

$table->dropForeign(['slack_user_id']);

Here is what the migration to change the foreign keys on my Cake model looked like.

DB::table('cakes')->get()->map(
	function ($cake) {
		$giver = DB::table('slack_users')->select('slack_user_id')->where('id', '=', $cake->giver_id)->get()->first();
		$giver = User::where('slack_user_id', '=', $giver->slack_user_id)->withTrashed()->first();
		$target = DB::table('slack_users')->select('slack_user_id')->where('id', '=', $cake->target_id)->get()->first();
		$target = User::where('slack_user_id', '=', $target->slack_user_id)->withTrashed()->first();
		DB::table('cakes')->where('id', $cake->id)->update(['giver_id' => $giver->id, 'target_id' => $target->id]);
	}
);

Again, notice that I’m not using Eloquent to access the SlackUser model, instead relying on the DB:class facade. I’m free to delete the SlackUser::class at anytime now!

All of this code was added to the up() method of my migration, and I carefully reversed all of the column changes for the down() method. One thing I did not do in the down method, was remigrate any data. I figured if the deploy went so bad that I needed to do that, then I would be better off restoring the database entirely from a backup instead. The down() changes I made were purely so I could migrate up/down for tests, which weren’t reliant on any database values anyway.

That brings me to another pain point: tests! 90% of my tests had to be updated since they mostly relied on the SlackUser. I started with one test file at a time, running the entire file first, then each failing test. I generally changed any instance of SlackUser to User first and then saw what broke. The first few were painful as there were references to relationships that needed to be updated. Often times while doing this, I would catch something that needed to be updated in my migration as well.

Eventually, most of the methods from SlackUser were migrated to User and all of the relationships were updated. Views were the last thing to be checked and I even managed to add a few missing tests based on failures I found while manually browsing the site locally.

// One of my newly added tests!
/** @test */
public function a_user_can_view_the_perk_redemption_page() //phpcs:ignore
{
	$this->withoutExceptionHandling();
	$user = factory(User::class)->create([
	  'is_owner' => true
	]);

	// Make fake users for the manager selection.
	factory(User::class, 5)->create([
	  'team_id' => $user->team->id
	]);

	$this->actingAs($user);

	$user->team->perks()->create([
	  'title' => 'an image perk',
	  'cost' => 100,
	  'image_url' => 'https://perk-image.com/perk.jpg'
	]);

	$this->assertCount(1, Perk::all());

	$res = $this->get(route('redeem-perk-create', Perk::first()));
	$res->assertOk();

	// Check for all of the names on the page. (manager selection for perk redemption)
	$res->assertSee([...User::all()->map->name]);
}

In the end, the Github PR had 57 changed files! A huge undertaking by any standard. I would venture a guess that those 57 files represent 80-90% of all of the code I had written for the app.

Overall, I’m happy I did this, as the logic surrounding users is much simpler to understand. I also got the opportunity to try out some new things and learn a bit more about the built in DB facade. I’m still kinda intimidated by SQL in general, but I’m getting more courageous every time I tackle one of these projects. Backups are still really important though! 😉

Conclusion

If you have any questions, feel free to reach out to me on Twitter or leave a comment!