Entries filed under PHP

Laravel QueueWorker: database not configured

Posted on 4. November 2019 Comments

It doesn’t matter which queue system you’re using: Redis, a relational database such as MySQL or a cloud solution like AWS SQS ; you can always store failed jobs in a normal database table. This configuration is done in config/queue.php in the last array:

'failed' => [
    'driver' => 'database',
    'database' => 'default', // NOTE: connection, not database!
    'table' => 'jobs_failed',
],

The key is to put in the connection name, not the database name for ‚database‘. If you don’t you will get the error message „Database default not configured“ every time a job fails instead.

Check out my Laravel Job Eloquent Models.


Laravel Dusk: The currently authenticated user is not who was expected.

Posted on 31. Oktober 2019 Comments

If you’re having trouble using $browser->loginAs($user) it might be because the SESSION_DRIVER is set to array. So even this simple code will fail:

$this->browse(function (Browser $browser) {
    $browser->loginAs(User::first())
            ->assertAuthenticatedAs(User::first());
});

Error: The currently authenticated user is not who was expected.
Failed asserting that two arrays are identical.

/vendor/laravel/dusk/src/Concerns/InteractsWithAuthentication.php:102
/tests/Browser/AuthTest.php:162
/vendor/laravel/dusk/src/Concerns/ProvidesBrowser.php:67
/tests/Browser/AuthTest.php:163

Write a .env.dusk.local (or where ever you’re starting the tests from) and use a persistent driver like SESSION_DRIVER=database or SESSION_DRIVER=file

Crawling with Web Storage in PHP

Posted on 28. September 2019 Comments

Some pages have an extra screen before the real site appears, e.g. for age verification (like porn, for now anyways, or drug sites) or country/currency selection (like travel and big company sites). As far as I researched this, sometimes the selection is stored in a cookie, sometimes in Web Storage.

Crawling with Cookies is no problem with Guzzle.

But for example, at https://cannabis.wiki the age verification is stored in the Local Storage as you can see in the Chrome DevTools:

image

Using spatie/crawler we can inject a Browsershot (puppeteer) instance, that sets the localStorage key ageConfirmed through custom JavaScript:

$js = "localStorage.setItem('ageConfirmed', '1');";
$browsershot->setOption('addScriptTag', json_encode(['content' => $js]));

However, the code is only injected after the side is already loaded. Therefore we have to use JavaScript to reload the page:

$js = "localStorage.setItem('ageConfirmed', '1');location.reload()";
$browsershot->setOption('addScriptTag', json_encode(['content' => $js]));

You can test this on the console e.g. by using ->save($pathToFile); and have a look at the screenshot to see if it worked.

Laravel: Serialization of ‚Closure‘ is not allowed with Jobs and Queues

Posted on 26. Februar 2018 Comments

When dispatching Jobs onto a queue it’s possible to get the following error:

Serialization of ‚Closure‘ is not allowed

This basically means that you are either passing a class that contains anonymous functions („closures“ in PHP) or that in the Job itself there is an anonymous function. This could also be a library you are using. PHP is unable to serialize closures per se. When Jobs get dispatched onto the queue, everything has to be serialized before. If you absolutely must use them this way, look into the Super Closure package by jeremeamia. However, just rewriting the code a little might be easier:

 

Solution

Instead of passing a whole object to the Job like so:

$objOfClassWithClosures = ...;
$job = new Job($objOfClassWithClosures);
dispatch($job);

Pass the Fully Qualified Class Name (that’s what ClassName::class returns) to the job and instantiate the object on the other side, possibly even using Laravels Container:

$job = new Job(ClassWithClosures::class);
dispatch($job);

Inside the job you can then use the constructor to make a new instance of that class:

public function __construct(string $fqcn) {
$objOfClassWithClosures = app()->make($fqcn);
}

 

PHP + Laravel development Packages for atom.io

Posted on 30. Januar 2018 Comments

PHP CS Fixer

brew install php-cs-fixer

apm install php-cs-fixer

Settings:

php path: /usr/local/bin/php

php-cs-fixer path: /usr/local/bin/php-cs-fixer

PHP CS Fixer Rules: @PSR2,blank_line_after_opening_tag,whitespace_after_comma_in_array,blank_line_after_namespace

PHP CS Fixer Arguments: --using-cache=no, --no-interaction, --stop-on-violation

[x] Show Notifications

Ctrl+Cmd+S

Can also be used on commandline for folders, e.g.
$ php-cs-fixer fix app/ --rules=@PSR2,blank_line_after_opening_tag,whitespace_after_comma_in_array,blank_line_after_namespace

 

php-integrator-refactoring (php-integrator-base)

apm install php-integrator-base

apm install php-integrator-refactoring

Linting (also see linter(-php))

Sort use statements

 

linter and linter-php

apm install linter

apm install linter-php

In case linting with php-integrator-base does not work.

 

language-blade and language-vue

apm install language-blade

apm install language-vue

 

minimap

apm install minimap

Short overview over the code on the right side

 

highlight-selected

apm install highlight-selected

 

platform-ide-terminal

apm install platform-ide-terminal

Instead of switching windows, terminal right in atom

Cmd+Shift+T

 

symbols-tree-view

apm install symbols-tree-view

Structured overview on the right side over classes, constants, attributes, methods etc

 

hyperclick and hyperclick-php

apm install hyperclick

apm install hyperclick-php

Cmd+Click on classes, functions etc to go to origin.

Not needed if you installed atom-ide-ui

 

teletype

apm install teletype

Work on the same file with coworkers, encrypted via WebRTC

 

More dependencies might be installed with installing some of these packages

PHP Commandline Notification

Posted on 3. Januar 2018 Comments

When you run long (PHP) jobs it’s easy to forget about the terminal. That’s why I’ve created a small PHP package that will make a little sound. Because PHP itself doesn’t have this functionality anymore it’s possible to just echo the ASCII sign for BEL. Simply put this at the end of your PHP job:

use repat\CommandlineBell;
// flashes screen if possible, otherwise just bell()
CommandlineBell::flash();
// makes a beep sound
CommandlineBell::bell();

Under the hood it’s really just:
echo "0x07";

A good (non-PHP) alternative I found is brb by Viktor Fröberg, just run commands like this

php artisan migrate --seed ; brb

Of course it’s always possible to just run a TTS app on the commandline via exec() like this:

exec("say terminal task done");

Or you could do it similar to brb:

php artisan migrate --seed ; say seeding done

Source on GitHub

Quick and dirty redirect script for the fitting shipping provider

Posted on 2. März 2017 Comments

Another way to use the PHP library shipping-service-provider-check apart from fixing the shipping providers in the ERP Plentymarkets is a simple quick and dirty PHP script that redirects to the tracking page of a fitting shipping provider. You can find the whole sourcecode on GitHub.

Installation

If you don’t use some sort of MVC this is a quick way to do it. If you do, I suspect you’ll know how to alter this example to fit your system. Feel free to write me an e-mail or comment if you need help

  1. Log onto your server and create a folder
  2. Copy or git pull the files, including .htaccess
  3. Get Composer
  4. composer require repat/shipping-service-provider-check

Explanation of the code

The $trackingId variable is received via HTTP GET. This means the URL would be http://url.tld/trackingscriptfolder/script.php?tracking_id=123456 (or without the script.php in case of the .htaccess), where 123456 is the TrackingID. The rest is more or less just for debugging. The scripts returns a HTTP Code 422 and ends if there is no input given.

$trackingId = $_GET["tracking_id"];
if (empty($trackingId)) {
  $unprocessableEntity = 422;
  http_response_code($unprocessableEntity);
  echo "wrong input";
  return;
}

Next the URLs will be defined. The keys have to match the shipping providers so make sure the shipping providers are added at the library as well. The TrackingID will be added at the end of the URL, so make sure they have the right format.

$shippingProviderURLs = [
  "dhl" => 'https://nolp.dhl.de/nextt-online-public/set_identcodes.do?idc=',
...
];

Then follows the documented way of checking for the providers. It could technically be that at the same time a TrackingID is valid for more than one provider. To keep things simple, we’ll just use the first one (array_search instead of array_keys). In case there is none found, array_search returns false and the script continues. In case there is one found, the header() function is called to redirect the user to the correct provider page.

$checkedProvider = array_search(true, $result);
$urlOfCheckedProvider = $shippingProviderURLs[$checkedProvider];

if ($checkedProvider !== false) {
  header('location: ' . $urlOfCheckedProvider . $trackingId);
}

Now it’s up to you what you will do with the ones you can’t redirect. I decided to write a little skeleton.html file and include that with a simple message that the user could try the shipping providers where a check is not possible (no API, no website scraping, no regex). They are listed earlier:

$shippingProviderURLsNoCheck = [
  "DPD" => 'https://tracking.dpd.de/parcelstatus?query=',
...

Fix shipping providers for FBA orders in Plentymarkets

Posted on 27. Februar 2017 Comments

The popular ERP Plentymarkes let’s you send your orders via the Fulfillment by Amazon program to your customers. Amazon is just used as a logistics partner, the order doesn’t have to come via Amazon. It could e.g. be a Rakuten or eBay order. The problem that Plentymarkets didn’t address at all and is discussed in the private forums, is that Amazon chooses whatever shipping company they seem fit.

Every order has a standard shipping provider and with it comes a URL for tracking the package. Most of the vendors would implement an event procedure („Ereignisaktion“) that sends an email to the customer containing the order and the TrackingID as soon as it’s shipped. The TrackingIDs are directly imported from Amazon. Unfortunately Plentymarkets does not change the shipping provider in case it’s not the default one (which is often the case). So the customer gets wrong information.

In one of last last blogposts I introduced a PHP package to check which shipping provider a TrackingID belongs to by either scraping their website or using a regular expression and filtering for common patterns. In this blog post I want to explain how to use it to fix those orders. You will need a webserver with PHP and MySQL and a little knowledge of programming and Linux.

You can find some simplified code examples on GitHub soon.

  1. Take your FBA orders out of the event procedures that sends the email to the customers, filter e.g. via WarehouseID
  2. Instead, use an event procedure to mark those orders with a flag, e.g. a star (ID 2)
  3. Create a DynamicExport of the type OrderComplete. Filter for your Sent Status (usually 7) and WarehouseID of FBA orders. Include these fields:
    1. OrderID
    2. OrderPackageNo
    3. OrderParcelServiceID
    4. OrderParcelServicePresetID
    5. OrderLastUpdateTimestamp
  4. Implement the SOAP Call GetDynamicExport (will be replaced by REST via Plugin mid-2017)
  5. Write a Cronjob that calls the GetDynamicExport script with the FormatID of the DynamicExport from #2
  6. Create a database table called ‚pm_shipping_providers_check‚ with the fields from #2
  7. Write a script that imports the CSV you downloaded with the GetDynamicExport call into your MySQL database
  8. Create another DynamicExport, this time of the type Order. Include these fields:
    1. OrderID (Synchronisation)
    2. ParcelServiceID (Import)
    3. ParcelServicePresetID (Import)
    4. PackageNo (Import)
  9. Create a table with these fields called ‚pm_shipping_providers_correction
  10. Create a script, that uses the shipping-service-provider-check library to check the every PackageNo in ‚pm_shipping_providers_check‘ and – if the provider is different from the default one – writes it into ‚pm_shipping_providers_correction‘
  11. Implement the SOAP Call SetDynamicImport (see #3)
  12. Write a script that exports the table ‚pm_shipping_providers_correction‘ into a (semicolon seperated) CSV.
  13. Create a cronjob that uploads the file regularly
  14. Create an event procedure on the event PackageNo that sends out emails, but only for the previously flagged orders

Herausfinden zu welchem Versanddienstleister eine Tracking ID gehört

Posted on 29. Juli 2016 Comments

Wir hatten erst letztens das Problem, dass wir automatisiert Tracking IDs zugespielt bekommen, die von mehreren Versanddienstleistern stammen könnten. Dies ist z. B. beim ERP Plentymarkets der Fall, wenn man Pakete per FBA verschickt. Allerdings nur, wenn die Aufträge nicht von Amazon kommen, sondern ein sog. Multichannel Auftrag vorliegt, also Amazon nur als Logistiker benutzt wird. In der API Rückmeldung steht zwar m.E. der Versanddienstleister, allerdings implementiert Plentymarkets kein Mapping.

Um den richtigen Provider zu finden, ohne jedes mal alle Websites abzuklappern habe ich das PHP Paket shipping-service-providers-check geschrieben und über composer verfügbar gemacht. Es macht automatisiert genau das. Falls es nicht geht, weil z.B. die Flash oder JavaScript im Spiel sind oder es gar keine öffentliche Seite gibt (->Amazon Logistics), kann die Nummer auch anhand des Formats geprüft werden, also mit einem regulärem Ausdruck.

Theoretisch ist es möglich, dass die Tracking ID gleichzeitig bei mehreren Providern für unterschiedliche Sendungen gültig ist. Das ist zugegebenermaßen extrem unwahrscheinlich. Dennoch gibt mein Paket ein Array von Paketdienstleistern, jeweils mit einem Boolean (true/false), zurück.

Code Erklärung

Die Abhängigkeiten sind fabpot/goutte (ein Website scraper) und danielstjules/stringy (für Stringvergleiche). Da ich von PHPs use function Gebrauch mache, ist PHP Version 5.6 notwendig.

Die Klasse Check enthält neben dem Konstruktur, der die TrackingID erwartet, 3 weitere Methoden.

  • getProviders() – gibt alle Provider zurück
  • checkAll($extraProviders) – gibt das eben erwähnte Array zurück. Das hier ist die eigentlich wichtige öffentliche Methode
  • check() – private, wird in einer Schleife von checkAll() aufgerufen

In der Datei default_providers.php sind in einem Array alle implementierten Versanddienstleister aufgeführt. Hier können auch weitere hinzugefügt werden, bzw. in diesem Format an checkAll() übergeben werden. Jeder Dienstleister hat 3 Parameter:

  • base_url – URL, die goutte zusammen mit der tracking ID aufrufen wird
  • filter – HTML Tag auf der Seite, nach dem gesucht bzw. das durchsucht werden soll
  • search_string  – nach diesem String wird in dem HTML Tag gesucht

Der eigentlich wichtige Teil ist deswegen die Methode checkOnline()

$crawler = $this->client->request('GET', $parameters["base_url"] . $this->trackingId);

in dieser Zeile ruft goutte die vorher definierte base_url mit der Tracking ID auf. Die URL muss deshalb im Format http://example.com?tracking_id= vorliegen. Aus der Tracking ID 123456 lautet dann der Aufruf http://example.com?tracking_id=123456.

$crawler->filter($parameters["filter"])->each(function ($node) use ($parameters) {
            if (s($node->text())->contains($parameters["search_string"])) {
                return true;
            }
            return false;

In dem HTML Tag aus dem filter Parameter wird jetzt nach dem String search_string mittels stringys contains() gesucht. Sollte das der Fall sein, wird true zurückgegeben. Da die Funktion each ein Array zurückgibt und es möglich ist, dass das relevante HTML Tag mehrmals vorkommt, wird danach geguckt ob true in diesem array überhaupt vorkommt, auch wenn andere Einträge in diesem Array eben false sind.

return in_array(true, ...)

Dies wird in einer Schleife in checkAll() durchgegangen:

        foreach ($shippingProviders as $shippingProvider => $parameters) {
            $response[$shippingProvider] = $this->check($parameters);
        }

Außerdem ist es durch folgende Zeile möglich einen Versanddienstleister hinzuzufügen oder zu ersetzen (beides passiert durch array_merge())

  if (isset($shippingProviders)) {
            $shippingProviders = array_merge($defaultShippingProviders, $shippingProviders);
        } else {
            $shippingProviders = $defaultShippingProviders;
        }

Die Methode zum Überprüfen des regulärem Ausdrucks checkFormat() ist denkbar simpel:

  boolval(preg_match($parameters["regex"], $this->trackingId))

Da preg_match 1 zurückgibt, falls der Regex zutrifft, und 0 falls nicht (false wenn ein Fehler aufgetreten ist), muss um das Ergebnis noch boolval().

Die Anleitung zum Benutzen des Pakets ist auf GitHub bzw. Packagist. Zur Zeit des Verfassens dieses Artikels sind die folgenden Versanddienstleister implementiert:

  • DHL
  • GLS
  • UPS
  • Hermes
  • Amazon Logistics

Fedex, DPD und TNT gestalten sich schwierig, da diese Informationen per JavaScript nachladen und goutte das nicht beherrscht. Ggf. werde ich nochmal ein npm Paket mit zombie.js und/oder phantomjs schreiben.

MySQL Workbench Error: line contains NULL byte

Posted on 21. April 2016 Comments

When importing CSV (or other) files into the database, scripts (especially PHP or C-related languages), will stop if there is – for whatever reason – a NULL byte in your file because it signals end of file/string, see Null bytes related issues.

So when importing a file like that with MySQL Workbench you will get this error:

line contains NULL byte

You can solve this by using the commandline tool tr (from coreutils):

tr < file-with-nulls -d '\000' > file-without-nulls

To check if there are any null bytes in your file, use the python IDE and type in:

open('filename.ext').read().index('\0')

Thanks to Pointy from Stackoverflow