Build APIs You Won't Hate: Everyone and their dog wants an API, so you should probably learn how to build them (2014)

6. Outputting Data

6.1 Introduction

In Chapter 3: Input and Output Theory we looked at the theory of the output structure and the pros and cons for various different formats. The rest of this book assumes you have picked your favorite, and it assumes that favorite is my favorite. This doesn’t matter all that much but doing everything for everyone would be an exercise in futility and boredom.

The aim of this chapter is to help you build out your controller endpoints. Assuming you have written tests for these endpoints before they exist, we can now fill up a few of those tests with green lights instead of the omnishambles of errors and fails you are most likely facing.

The examples in the first section will be trying to show off a list of places, and show of one specific place:

 1 {

 2     "data": [

 3         {

 4             "id": 2,

 5             "name": "Videology",

 6             "lat": 40.713857,

 7             "lon": -73.961936,

 8             "created_at": "2013-04-02"

 9         },

10         {

11             "id": 1,

12             "name": "Barcade",

13             "lat": 40.712017,

14             "lon": -73.950995,

15             "created_at": "2012-09-23"

16         }

17     ]

18 }

1 {

2     "data": [

3         "id": 2,

4         "name": "Videology",

5         "lat": 40.713857,

6         "lon": -73.961936,

7         "created_at": "2013-04-02"

8     ]

9 }

6.2 The Direct Approach

The first thing that every developer tries to do is take their favorite ORM, ODM, DataMapper or Query Builder, pull up a query and wang that result directly into the output.

Dangerously bad example of passing data from the database directly as output


 1 <?php

 2 classPlaceControllerextends ApiController

 3 {

 4     publicfunction show($id)

 5     {

 6         return json_encode([

 7             'data' => Place::find($id)->toArray(),

 8         ]);

 9     }

10

11     publicfunction index()

12     {

13         return json_encode([

14             'data' => Place::all()->toArray(),

15         ]);

16     }

17 }


This is the absolute worst idea you could have for enough reasons for me to fill up a chapter on its own, but I will try to keep it to just a section.

warning

ORMs in Controllers

Your controller should definitely not have this sort of ORM/Query Builder logic scattered around the methods. This is done to keep the example to one class.

Performance: If you return “all” items then that will be fine during development, but suck when you have a thousand records in that table… or a million.

Display: PHP’s popular SQL extensions all type-cast all data coming out of a query as a string, so if you have a MySQL “boolean” field (generally this is a tinyint(1) field with a value of 0 or 1) will display in the JSON output as a string, with a value of "0" or "1" which is lunacy. If you’re using PostgreSQL it is even worse, the value directly output by PHP’s PostgreSQL driver is "f" or "t". Your mobile developers won’t like it one bit, and anyone looking at your public API is going to immediately consider this an amateur API. You want true or false as an actual JSON boolean, not a numeric string or a char(1).

Security: Outputting all fields can lead to API clients (users of all sorts) being able to view your users passwords, see sensitive information like email addresses for businesses involved (venues, partners, events, etc), gain access to secret keys and tokens generally not allowed. If you leak your forgotten password tokens for example then you’re going to have an EXTREMELY bad time, its as bad as leaking the password itself.

Some ORM’s have a “hidden” option to hide specific fields from being output. If you can promise that you and every single other developer on your team (now, next year and for the entire lifetime of this application) will remember about that then congratulations, you could also achieve world peace with a team that focused.

Stability: If you change the name of a database field, or modify your MongoDB document, or change the statuses available for a field between v3 and v4 then your API will continue to behave perfectly, but all of your iPhone users are going to have busted crashing applications and it is your fault. You will promise yourself that you won’t change things, but you absolutely will. Change happens.

So, next our theoretical developer friend will try hard-coding the output.

Laborious example of type-casting and formatting data for output


 1 <?php

 2 classPlaceControllerextends ApiController

 3 {

 4     publicfunction show($id)

 5     {

 6         $place = Place::find($id);

 7

 8         return json_encode([

 9             'data' => [

10                 'id'         => (int) $place->id,

11                 'name'       => $place->name,

12                 'lat'        => (float) $place->lat,

13                 'lon'        => (float) $place->lon,

14                 'created_at' => (string) $place->created_at,

15             ],

16         ]);

17     }

18

19     publicfunction index()

20     {

21         $places = array();

22

23         foreach (Place::all() as $place) {

24             $places[] = [

25                 'id'         => (int) $place->id,

26                 'name'       => $place->name,

27                 'lat'        => (float) $place->lat,

28                 'lon'        => (float) $place->lon,

29                 'created_at' => (string) $place->created_at,

30             ];

31         }

32

33         return json_encode([

34             'data' => $places,

35         ]);

36     }

37 }


Thanks to specifying exactly what fields to return in the JSON array the security issues are taken care of. The type-casting of various fields turn numeric strings into integers, coordinates into floats, and that pesky Carbon (DateTime) object from Laravel into a string, instead of letting the object turn itself into an array.

The only issue this has not taken care of from the above example is performance, but that is a job for pagination which will be covered in Chapter 10.

A new issue has however been created, which should be a fairly obvious one: This is icky. Our theoretical developer now tries something else.

Considerably better approach to formatting data for output


 1 <?php

 2 classPlaceControllerextends ApiController

 3 {

 4     publicfunction show($id)

 5     {

 6         $place = Place::find($id);

 7

 8         return json_encode([

 9             'data' => $this->transformPlaceToJson($place),

10         ]);

11     }

12

13     publicfunction index()

14     {

15         $places = array();

16         foreach (Place::all() as $place) {

17             $places[] = $this->transformPlaceToJson($place);

18         }

19

20         return json_encode([

21             'data' => $places,

22         ]);

23     }

24

25     privatefunction transformPlaceToJson(Place $place)

26     {

27         return [

28             'id'         => (int) $place->id,

29             'name'       => $place->name,

30             'lat'        => (float) $place->lat,

31             'lon'        => (float) $place->lon,

32             'created_at' => (string) $place->created_at,

33         ];

34     }

35 }


Certainly much better, but what if a different controller wants to show a place at any point? You could theoretically move all of these transform methods to a new class or shove them in the ApiController, but that would just be odd.

Really you want to make what I have come to call “Transformers”, partially because the name is awesome and because that is what they are doing.

These are essentially just classes which have a transform method, which does the same as the transformPlaceToJson() above, but to avoid you having to learn how to make your own I have released a PHP package which takes care of it: Fractal.

6.3 Transformations with Fractal

With Fractal, Transformers are created as either a callback or an instance of an object implementing League\Fractal\TransformerAbstract. They do exactly the job that our transformPlaceToJson() method did but they live on their own, are easily unit-testable (if that floats your boat) and remove a lot of presentation clutter from the controller.

Fractal does a lot more than that which will be explored later on, but it covers concerns with transformation perfectly, removes the security, stability and display concerns addressed earlier.

While other languages have great solutions for this already, PHP seemed to be rather lacking for this exact purpose. Some call it “Data Marshalling” or “Nested Serialization”, but it is all achieving roughly the same goal: take potentially complicated data from a range of stores and turn it into a consistent output.

·        Jbuilder looks fairly slick for the Ruby crowd

·        Tweet other suggestions to @philsturgeon

That is the end of theory in this book. We will now be working with code. Open up the Sample Code ZIP file or head to the GitHub repo and extract it somewhere useful.

1 $ cd chapter6

2 $ php artisan serve

3 Laravel development server started on http://localhost:8000

Open your browser and go to http://localhost:8000/places, and there is a list of places looking like this:

Fractal default JSON structure using the JSONView extension for Chrome

Fractal default JSON structure using the JSONView extension for Chrome

This is a Laravel 4 application but only because it has migrations and seeding and I like it. This is made up of a few bits of PHP that would work in any framework, and the approach works in any language.

·        composer.json - Added an autoloadable folder using PSR-0 to allow my own code to be loaded

·        app/controllers/ApiController.php - Insanely simple base controller for wrapping responses

·        app/controllers/PlaceController.php - Grab some data and pass it to the ApiController

Other than defining some basic GET routes in app/routes.php that is basically all that is being done.

The PlaceController looks like this:

Example of a controller using Fractal to output data


 1 <?php

 2 use App\Transformer\PlaceTransformer;

 3

 4 classPlaceControllerextends ApiController

 5 {

 6     publicfunction index()

 7     {

 8         $places = Place::take(10)->get();

 9         return $this->respondWithCollection($places, new PlaceTransformer);

10     }

11

12     publicfunction show($id)

13     {

14         $place = Place::find($id);

15         return $this->respondWithItem($place, new PlaceTransformer);

16     }

17 }


The “raw data” (happens to be an ORM model but could be anything) is sent back with the appropriate convenience method and a transformer instance is provided too. These respondWithCollection() and respondWithItem() methods come from ApiController, and their job is just to create Fractal instances without exposing as many classes to interact with.

The PlaceTransformer looks like this:

 1 <?php namespace App\Transformer;

 2

 3 use Place;

 4 use League\Fractal\TransformerAbstract;

 5

 6 classPlaceTransformerextends TransformerAbstract

 7 {

 8     /**

 9      * Turn this item object into a generic array

10      *

11      * @return array

12      */

13     publicfunction transform(Place $place)

14     {

15         return [

16             'id'           => (int) $place->id,

17             'name'         => $place->name,

18             'lat'          => (float) $place->lat,

19             'lon'          => (float) $place->lon,

20             'address1'     => $place->address1,

21             'address2'     => $place->address2,

22             'city'         => $place->city,

23             'state'        => $place->state,

24             'zip'          => (float) $place->zip,

25             'website'      => $place->website,

26             'phone'        => $place->phone,

27         ];

28     }

29 }

Simple.

The ApiController is kept super simple at this point too:

Simple ApiController for basic responses using Fractal


 1 <?php

 2

 3 use League\Fractal\Resource\Collection;

 4 use League\Fractal\Resource\Item;

 5 use League\Fractal\Manager;

 6

 7 classApiControllerextends Controller

 8 {

 9     protected $statusCode = 200;

10

11     publicfunction __construct(Manager $fractal)

12     {

13         $this->fractal = $fractal;

14     }

15

16     publicfunction getStatusCode()

17     {

18         return $this->statusCode;

19     }

20

21     publicfunction setStatusCode($statusCode)

22     {

23         $this->statusCode = $statusCode;

24         return $this;

25     }

26

27     protectedfunction respondWithItem($item, $callback)

28     {

29         $resource = new Item($item, $callback);

30

31         $rootScope = $this->fractal->createData($resource);

32

33         return $this->respondWithArray($rootScope->toArray());

34     }

35

36     protectedfunction respondWithCollection($collection, $callback)

37     {

38         $resource = new Collection($collection, $callback);

39

40         $rootScope = $this->fractal->createData($resource);

41

42         return $this->respondWithArray($rootScope->toArray());

43     }

44

45     protectedfunction respondWithArray(array $array, array $headers = [])

46     {

47         return Response::json($array, $this->statusCode, $headers);

48     }

49

50 }


The method respondWithArray() takes a general array to convert into JSON, which will prove useful with errors. Other than that everything you return will be a Fractal Item, or a Collection.

6.4 Hiding Schema Updates

Schema updates happen, and they can be hard to avoid. If the change in question is simply a renamed field then this is insanely easy to handle:

Before

1         'website' => $place->website,

After

1         'website' => $place->url,

By changing the right (our internal data structure) and keeping the left the same (the external field name) we maintain control over the stability for the client applications.

Sometimes it is a status change. A new status is added, or the change is fairly drastic and the status all change, but the old API version is still expecting the old one. Maybe someone changed “available” to “active” to be consistent with the other tables, because the original developer was as consistent and logical as a rabid ferret.

Before

1         'status' => $place->status,

After

1         'status' => $place->status === 'available' ? 'active' : $place->status,

Gross, but useful.

6.5 Outputting Errors

Exactly how to output errors is something I personally am still toying with. The current front-runner is adding convenience methods to the ApiController which handle global routes with a constant as the code and a HTTP error code set, with an optional message in case I want to override the message.

Simple error codes and responses added to ApiController


 1 <?php

 2

 3 // ...

 4

 5 classApiControllerextends Controller

 6 {

 7     // ...

 8

 9     const CODE_WRONG_ARGS = 'GEN-FUBARGS';

10     const CODE_NOT_FOUND = 'GEN-LIKETHEWIND';

11     const CODE_INTERNAL_ERROR = 'GEN-AAAGGH';

12     const CODE_UNAUTHORIZED = 'GEN-MAYBGTFO';

13     const CODE_FORBIDDEN = 'GEN-GTFO';

14

15      // ...

16

17     protectedfunction respondWithError($message, $errorCode)

18     {

19         if ($this->statusCode === 200) {

20             trigger_error(

21                 "You better have a really good reason for erroring on a 200...",

22                 E_USER_WARNING

23             );

24         }

25

26         return $this->respondWithArray([

27             'error' => [

28                 'code' => $errorCode,

29                 'http_code' => $this->statusCode,

30                 'message' => $message,

31             ]

32         ]);

33     }

34

35     /**

36      * Generates a Response with a 403 HTTP header and a given message.

37      *

38      * @return  Response

39      */

40     publicfunction errorForbidden($message = 'Forbidden')

41     {

42         return $this->setStatusCode(403)->respondWithError($message, self::CODE_FORBIDDEN);

43     }

44

45     /**

46      * Generates a Response with a 500 HTTP header and a given message.

47      *

48      * @return  Response

49      */

50     publicfunction errorInternalError($message = 'Internal Error')

51     {

52         return $this->setStatusCode(500)->respondWithError($message, self::CODE_INTERNAL_E\

53 RROR);

54     }

55

56     /**

57      * Generates a Response with a 404 HTTP header and a given message.

58      *

59      * @return  Response

60      */

61     publicfunction errorNotFound($message = 'Resource Not Found')

62     {

63         return $this->setStatusCode(404)->respondWithError($message, self::CODE_NOT_FOUND);

64     }

65

66     /**

67      * Generates a Response with a 401 HTTP header and a given message.

68      *

69      * @return  Response

70      */

71     publicfunction errorUnauthorized($message = 'Unauthorized')

72     {

73         return $this->setStatusCode(401)->respondWithError($message, self::CODE_UNAUTHORIZ\

74 ED);

75     }

76

77     /**

78      * Generates a Response with a 400 HTTP header and a given message.

79      *

80      * @return  Response

81      */

82     publicfunction errorWrongArgs($message = 'Wrong Arguments')

83     {

84         return $this->setStatusCode(400)->respondWithError($message, self::CODE_WRONG_ARGS\

85 );

86     }


This basically allows for generic error messages to be returned in your controller without having to think too much about the specifics.

Controller using Fractal, combined with a simple error response


 1 <?php

 2 use App\Transformer\PlaceTransformer;

 3

 4 classPlaceControllerextends ApiController

 5 {

 6     publicfunction index()

 7     {

 8         $places = Place::take(10)->get();

 9         return $this->respondWithCollection($places, new PlaceTransformer);

10     }

11

12     publicfunction show($id)

13     {

14         $place = Place::find($id);

15

16         if (! $place) {

17             return $this->errorNotFound('Did you just invent an ID and try loading a place? M\

18 uppet.');

19             }

20

21         return $this->respondWithItem($place, new PlaceTransformer);

22     }

23 }


Other “Place” specific errors could go directly into the PlaceController as methods just like these, with their own constants in the controller, picking a statusCode in the method or relying on one as an argument.

6.6 Testing this Output

You have already seen how to test your endpoints using the Gherkin syntax in Chapter 5: Endpoint Testing, so we can apply that testing logic to this output:

 1 Feature: Places

 2

 3 Scenario: Listing places without search criteria is not possible

 4     When I request "GET /places"

 5     Then I get a "400" response

 6

 7 Scenario: Finding a specific place

 8     When I request "GET /places/1"

 9     Then I get a "200" response

10     And scope into the "data" property

11         And the properties exist:

12             """

13             id

14             name

15             lat

16             lon

17             address1

18             address2

19             city

20             state

21             zip

22             website

23             phone

24             created_at

25             """

26         And the "id" property is an integer

27

28 Scenario: Searching non-existent place

29     When I request "GET /places?q=c800e42c377881f8202e7dae509cf9a516d4eb59&lat=1&lon=1"

30     Then I get a "200" response

31     And the "data" property contains 0 items

32

33

34 Scenario: Searching places with filters

35     When I request "GET /places?lat=40.76855&lon=-73.9945&q=cheese"

36     Then I get a "200" response

37     And the "data" property is an array

38     And scope into the first "data" property

39         And the properties exist:

40             """

41             id

42             name

43             lat

44             lon

45             address1

46             address2

47             city

48             state

49             zip

50             website

51             phone

52             created_at

53             """

54     And reset scope

This is again using the FeatureContext.php provided in the sample code, which makes it really easy to test output. We are again assuming that all output is in a "data" element, which is either an object (when one resource has been requested) or an array of objects (multiple resources or a collection have been requested).

When you are searching for data you want to ensure that not finding any data doesn’t explode. This can be down to your controller processing on output and failing because what should be an array is null, or because some PHP collection class is missing methods, etc. This is why we perform the search with a hardcoded invalid search term, then check that it returns an empty collection:

1 {

2     "data": []

3 }

The line And the "data" property contains 0 items will cover this. Then we can search for valid terms, knowing that our database seeder has made sure at least one Place has the keyword “cheese” in the name. Using the line And scope into the first "data" property the scope changes to be inside the first data item returned, and the properties can be checked for existence too. If no data, or required fields are missing, this test will fail.

6.7 Homework

Your homework is to take apart the sample application, fit it into your API and try to build valid output for as many of your GET endpoints as possible. Check the data types and make sure the array structure is being output in the way you expect using the test example above.

With valid output covered and basic errors covered, what is next? The most complicated part of API generation, which at some point every developer has to try and work out: embedding/nesting resources, or making “relationships”.