Data storage

Horváth Győző
Associate professor
1117 Budapest, Pázmány Péter sétány 1/c., 2.408-as szoba
Tel: (1) 372-2500/8469
horvath.gyozo@inf.elte.hu

Recap

Recap

Server-side web programming: a program (PHP) generates the content (HTML)

• Output
  • static prototype
  • dynamic template
• Input
  • client: link, form
  • PHP: $_GET, $_POST
• Form processing
  • Input, validation ($_POST, $_GET)
  • Data (no $_POST, $_GET, HTML)
  • Output (HTML)

Data from client to PHP

Server program input data

Data storage

Common data as a shared resource

Szkriptbeli változók

• They live during the script
• After it they are deleted

if (isset($counter)) {
  $counter += 1;
} else {
  $counter = 0;
}
var_dump($counter);

External data storage

File handling in PHP

Low-level file operations

// $mode
r (read only)
r+ (read, write)
w (write only, empty file)
w+ (read, write, empty file)
a (append)
a+ (append, read)
x (write only, empty file, if it exists, then error)
x+ (read, write, empty file, if it exists, then error)
c (write only, if it exists, then position to the beginning)
c+ (write, read, if it exists, then position to the beginning)

Low-level read/write

High-level read/write

• Read
  • $arr = file($filename[, modifiers])
    • each row an array element
  • $s = file_get_contents($filename)
    • whole file as a string
  • readfile($filename)
    • writing the file to the output (e.g. image)
  • fpassthru($f)
    • writing an already opened file to the output (e.g. image)
• Write
  • file_put_contents($filename, $s)
    • write string into a file

További fájlműveletek

Further file operations

• basename($path)
  • returning the filename
• chown($file, $user)
• chmod($file, $mode)
• chgrp($file, $group)
• stat($file)
  • file properties in an array
• fseek($f, $offset)
  • positioning file pointer

Error handling

• Error → returning false
• @ operator: suppressing error message

$f = @fopen('non_existing.txt', 'r');
if ($f) {
  /* ... */
  fclose($f);
}

File structure driven file storage

Use cases

• Fixed file structure
  • microformat
  • transparency
  • interoperability
• → with high and low level file operations

1. example

Film titles are given in a file. Read it into an array!

$films = file('lista.txt', 
               FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
/*
Array
(
    [0] => The hobbit
    [1] => The Lord of the Rings
    [2] => Into the wild
    [3] => The Passion of Christ
)
*/

The hobbit
The Lord of the Rings
Into the wild
The Passion of Christ
    

1. example – error handling

// Required
$films = @file('non_existing.txt', 
                FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES) 
          or die('File not found');

or

// Default data
$films = @file('non_existing.txt', 
                FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
if (!$films) {
  $films = array();
}

2. example

Save an array of film titles to an array!

// Array with film titles
$films = array(
  'Vuk',
  'Lolka és Bolka',
  'Macskafogó',
  'Kisvakond és barátai',
);

// Writing with low-level operations
define('ENDLINE', "\n");
$f = @fopen('tales.txt', 'w') 
     or die('Error!');
if ($f) {
  foreach ($films as $film) {
    fputs($f, $film . ENDLINE);
  }
  fclose($f);
}

// OR
define('ENDLINE', "\n");
// Elemek összefűzése
$s = implode(ENDLINE, $films) 
       . ENDLINE;
// Kiírás magas szintű művelettel
$success = @file_put_contents(
  'tales.txt', $s);

3. example

An array of records is given. Save it into a file, one record should go into one line, fields should be separated by tabulator sign.

$films = [
  [
    'title'     => 'The Passion of Christ',
    'director' => 'Mel Gibson',
    'year'      => '2004',
  ],
  [
    'title'     => 'Pio atya - A csodák embere',
    'director' => 'Carlo Carlei',
    'year'      => '2000',
  ],
];

3. example

define('ENDLINE', "\n");
$f = @fopen('films.txt', 'w')
     or die('Error!');
if ($f) {
  foreach ($films as $film) {
    fputcsv($f, $film, "\t");
  }
  fclose($f);
}

$films = array(
  array(
    'title'     => 'The Passion of Christ',
    'director' => 'Mel Gibson',
    'year'      => '2004',
  ),
  array(
    'title'     => 'Pio atya - A csodák embere',
    'director' => 'Carlo Carlei',
    'year'      => '2000',
  ),
);

The Passion of Christ  Mel Gibson  2004
Pio atya - A csodák embere  Carlo Carlei    2000
    

4. example

Load the file from the previous example as an array of records!

The Passion of Christ  Mel Gibson  2004
Pio atya - A csodák embere  Carlo Carlei    2000
    

// Reading with the file function
$films = file('films.txt', 
               FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
foreach($films as &$film) {
  list($c, $r, $e) = explode("\t", $film);
  $film = array(
    'title'     => $c,
    'director' => $r,
    'year'      => $e,
  );
} 

4. example

Load the file from the previous example as an array of records!

The Passion of Christ  Mel Gibson  2004
Pio atya - A csodák embere  Carlo Carlei    2000
    

//Beolvasás soronként az fgetcsv-vel 
$films = array();
$f = @fopen('films.txt', 'r');
if ($f) {
  while ($sor = fgetcsv($f, 0, "\t")) {
    $films[] = array(
      'title'     => $sor[0],
      'director' => $sor[1],
      'year'      => $sor[2],
    );
  }
  fclose($f);
}

5. example

• Save an array of records into a file. Every field should go into separate row. The records are separated by an empty line.
• Solve the reading as well!
• Low-level file operations

5. example – output

// Save to file
define('ENDLINE', "\n");
$f = @fopen('films_tobbsor.txt', 'w')
     or die('Error!');
if ($f) {
  foreach ($films as $film) {
    fputs($f, $film['title'] . ENDLINE);
    fputs($f, $film['director'] . ENDLINE);
    fputs($f, $film['year'] . ENDLINE);
    fputs($f, ENDLINE);
  }
  fclose($f);
}

$films = array(
  array(
    'title'     => 'The Passion of Christ',
    'director' => 'Mel Gibson',
    'year'      => '2004',
  ),
  // ...
);

The Passion of Christ
Mel Gibson
2004

Pio atya - A csodák embere
Carlo Carlei
2000
    

5. example – loading

The Passion of Christ
Mel Gibson
2004

Pio atya - A csodák embere
Carlo Carlei
2000
    

// Load from file
$films = array();
$f = @fopen('films_multiline.txt', 'r');
if ($f) {
  while (!feof($f)) {
    $title = trim(fgets($f));
    $director = trim(fgets($f));
    $year = trim(fgets($f));
    $empty = fgets($f);
    if ($title != '') {
      $films[] = array(
        'title'       => $title,
        'director'   => $director,
        'year'        => $year,
      );
    }
  }
  fclose($f);
}

Concurrent file usage

define('ENDLINE', "\n");
$f = @fopen('tales.txt', 'w')
     or die('Error!');
if ($f) {
  if (flock($f, LOCK_EX)) {
    foreach ($films as $film) {
      fputs($f, $film . ENDLINE);
    }
    flock($f, LOCK_UN);
  }
  fclose($f);
}

Concurrent file usage

$films = array();
$f = @fopen('list.txt', 'r');
if ($f) {
  if (flock($f, LOCK_SH)) {
    while (!feof($f)) {
      $sor = trim(fgets($f));
      if ($sor != '') {
        $films[] = $sor;
      }
    }
    flock($f, LOCK_UN);
  }
  fclose($f);
}

Data structure driven file storage

Data structure

• Saving and loading data structure
• File content does not matter
• Data availability is important
• Data structure → text format
• → serialization

Serialization

• A reversible textual equivalent of a data structure.
• Usage
  • persistence
  • transmission
• Functions for serializing
  • serialize($value) → string
  • unserialize($string) → value
  • json_encode($value) → string
  • json_decode($string) → value

Example

$films = [
  [
    'title'     => 'The Passion of Christ',
    'director'  => 'Mel Gibson',
    'year'      => '2004',
    'actors' => [
      'Jim Caviezel',
      'Maia Morgenstern',
      'Christo Jivkov',
    ],
  ],
  [
    'title'     => 'Feltámadás',
    'director'  => 'Kevin Reynolds',
    'year'      => '2016',
    'actors' => [
      'Joseph Fiennes',
      'Tom Felton',
      'Cliff Curtis',
    ],
  ],
];

Serialize, unserialize

$s = serialize($films);
//a:2:{i:0;a:4:{s:5:"title";s:21:"The Passion of Christ";s:8:"director";s:10:"Mel Gibson";s:4:"year";s:4:"2004";s:6:"actors";a:3:{i:0;s:12:"Jim Caviezel";i:1;s:16:"Maia Morgenstern";i:2;s:14:"Christo Jivkov";}}i:1;a:4:{s:5:"title";s:12:"Feltámadás";s:8:"director";s:14:"Kevin Reynolds";s:4:"year";s:4:"2016";s:6:"actors";a:3:{i:0;s:14:"Joseph Fiennes";i:1;s:10:"Tom Felton";i:2;s:12:"Cliff Curtis";}}}
$films2 = unserialize($s);
/*
Array
(
  [0] => Array
    (
      [title] => The Passion of Christ
      [director] => Mel Gibson
      [year] => 2004
      [actors] => Array
        (
          [0] => Jim Caviezel
          [1] => Maia Morgenstern
          [2] => Christo Jivkov
        )
    )
  [1] => Array
    (
      [title] => Feltámadás
      [director] => Kevin Reynolds
      [year] => 2016
      [actors] => Array
        (
          [0] => Joseph Fiennes
          [1] => Tom Felton
          [2] => Cliff Curtis
        )
    )
)*/

json_encode

$s = json_encode($films);
//[{"title":"The Passion of Christ","director":"Mel Gibson","year":"2004","actors":["Jim Caviezel","Maia Morgenstern","Christo Jivkov"]},{"title":"Felt\u00e1mad\u00e1s","director":"Kevin Reynolds","year":"2016","actors":["Joseph Fiennes","Tom Felton","Cliff Curtis"]}]

[
  {
    "title": "The Passion of Christ",
    "director": "Mel Gibson",
    "year": "2004",
    "actors": [
      "Jim Caviezel",
      "Maia Morgenstern",
      "Christo Jivkov"
    ]
  },
  {
    "title": "Felt\u00e1mad\u00e1s",
    "director": "Kevin Reynolds",
    "year": "2016",
    "actors": [
      "Joseph Fiennes",
      "Tom Felton",
      "Cliff Curtis"
    ]
  }
]

json_decode

$films = json_decode($s, true);
/*
Array
(
  [0] => Array
    (
      [title] => The Passion of Christ
      [director] => Mel Gibson
      [year] => 2004
      [actors] => Array
        (
          [0] => Jim Caviezel
          [1] => Maia Morgenstern
          [2] => Christo Jivkov
        )
    )
  [1] => Array
    (
      [title] => Feltámadás
      [director] => Kevin Reynolds
      [year] => 2016
      [actors] => Array
        (
          [0] => Joseph Fiennes
          [1] => Tom Felton
          [2] => Cliff Curtis
        )
    )
)*/

json_decode

$films = json_decode($s, false);
/*
Array
(
  [0] => stdClass Object
    (
      [title] => The Passion of Christ
      [director] => Mel Gibson
      [year] => 2004
      [actors] => Array
        (
          [0] => Jim Caviezel
          [1] => Maia Morgenstern
          [2] => Christo Jivkov
        )
    )
  [1] => stdClass Object
    (
      [title] => Feltámadás
      [director] => Kevin Reynolds
      [year] => 2016
      [actors] => Array
        (
          [0] => Joseph Fiennes
          [1] => Tom Felton
          [2] => Cliff Curtis
        )
   )
)*/

Helper functions

function load_from_file(string $filename, bool $array_result = false, $default_data = []) {
  $s = @file_get_contents($filename);
  return ($s === false 
    ? $default_data 
    : json_decode($s, $array_result));
}

function save_to_file(string $filename, $data) {
  $s = json_encode($data);
  return file_put_contents($filename, $s, LOCK_EX);
}

For any data structure

Example – adding a new actress

$films = load_from_file('films.txt');

$films[0]['actors'][] = 'Monica Bellucci';

save_to_file('films.txt', $films);

[
  {"title":"The Passion of Christ","director":"Mel Gibson","year":"2004",
   "actors":["Jim Caviezel","Maia Morgenstern","Christo Jivkov","Monica Bellucci"]},
  {"title":"Felt\u00e1mad\u00e1s","director":"Kevin Reynolds","year":"2016",
   "actors":["Joseph Fiennes","Tom Felton","Cliff Curtis"]}
]

Disadvantages

• Moving a lot of data
• Bad concurrency handling
• Manual filtering the result

Data processing

Array operations

• array_map(callable $fn, array $x)
• array_filter(array $x, callable $fn)
• array_reduce(array $x, callable $fn, $initial)
• array_sum(array $x)
• array_walk(array $x, callable $fn)

$numbers = [1, 2, 3, 4, 5];
$evens = array_filter($numbers, function ($e) {
  return $e % 2 === 0;
});

Array operations

• in_array($what, $arr): decision
• array_search($what, $arr): linear search
• sort($arr)
• shuffle($arr)
• array_keys($arr)
• array_values($arr)
• array_key_exists($kulcs, $arr)

Example

Let us show a list of movies! Put a filter box above the list!

<?php
// Data
function getMoviesByYear(int $year = NULL): array {
  // ...
}
// Main
$year = (int)$_GET["year_filter"] ?? NULL;

$movies = getMoviesByYear($year);
?>
<h1>Movies</h1>
<form action="">
  <input name="year_filter">
  <button>Filter</button>
</form>
<ul>
  <?php foreach ($movies as $movie) : ?>
    <li>
      <?= $movie["title"] ?>
      (<?= $movie["year"] ?>)
    </li> 
  <?php endforeach ?>
</ul>

Data

movies.json

{
  "1": {
    "title": "The Passion of the Christ",
    "year": 2004,
    "director": "Mel Gibson"
  },
  "2": {
    "title": "Risen",
    "year": 2016,
    "director": "Kevin Reynolds"
  }
}

Example

function getMoviesByYear(int $year = NULL): array {
  $all = load_from_file('movies.json', true);
  return array_filter($all, function ($movie) use ($year) {
    return $movie['year'] === $year;
  });
}

$movies = getMoviesByYear(2004);

JsonIO

interface FileIO {
  public function save_to_file(string $filename, $data);
  public function load_from_file(string $filename, bool $array_result = false, $default_data = []);
};
class JsonIO implements FileIO {
  public function save_to_file(string $filename, $data) {
    $s = json_encode($data);
    return file_put_contents($filename, $s, LOCK_EX);
  }
  public function load_from_file(string $filename, bool $array_result = false, $default_data = []) {
    $s = @file_get_contents($filename);
    return ($s === false 
      ? $default_data 
      : json_decode($s, $array_result));
  }
}

JsonStorage (array)

class JsonStorage {
  private $io;
  private $filename;
  public function __construct(string $filename) {
    $this->io = new JsonIO();
    $this->filename = $filename;
  }
  public function all() {
    return $this->io->load_from_file($this->filename, true);
  }
  public function filter(callable $fn) {
    $all = $this->all();
    return array_filter($all, $fn);
  }
  public function insert(array $item): string {
    $all = $this->all();
    $id = uniqid('', true);
    $item['_id'] = $id;
    $all[$id] = $item;
    $this->io->save_to_file($this->filename, $all);
    return $id;
  }
  public function update(callable $filter, callable $updater) {
    $all = $this->all();
    array_walk($all, function (&$item) use ($filter, $updater) {
      if ($filter($item)) {
        $updater($item);
      }
    });
    $this->io->save_to_file($this->filename, $all);
  }
  public function delete(callable $filter) {
    $remaining = $this->filter(function ($elem) use ($filter) {
      return !$filter($elem);
    });
    $this->io->save_to_file($this->filename, $remaining);
  }
}

JsonStorage (array)

function getMoviesByYear(int $year = NULL): array {
  $movieRepository = new JsonStorage('movies.json');
  return $movieRepository->filter(function ($movie) use ($year) {
    return $movie['year'] === $year;
  });
}
function changeDirectorName(string $old, string $new) {
  $movieRepository = new JsonStorage('movies.json');
  $movieRepository->update(
    function ($movie) use ($old) {
      return $movie['director'] === $old;
    },
    function (&$movie) use ($new) {
      return $movie['director'] = $new;
    }
  );
}

JsonStorage (object)

function getMoviesByYear(int $year = NULL): array {
  $movieRepository = new JsonStorage('movies.json');
  return $movieRepository->filter(function ($movie) use ($year) {
    return $movie->year === $year;
  });
}

class JsonStorage {
  // ...
  public function all() {
    return (array)$this->io->load_from_file($this->filename, false);
  }
  public function insert(object $item): string {
    $all = $this->all();
    $id = uniqid('', true);
    $item->_id = $id;
    $all[$id] = $item;
    $this->io->save_to_file($this->filename, $all);
    return $id;
  }
  // ...
}

Repository

function getMoviesByYear(int $year = NULL): array {
  $movieRepository = new MovieRepository();
  return $movieRepository->filter(function ($movie) use ($year) {
    return $movie->year === $year;
  });
}
// OR
$movies = $movieRepository->getMoviesByYear(2016);

class MovieRepository extends JsonStorage {
  public function __construct() {
    parent::__construct('movies.json');
  }
  public function getMoviesByYear(int $year = NULL): array {
    return $this->filter(function ($movie) use ($year) {
      return $movie->year === $year;
    });
  }
}

Repository (type safe)

$movieRepository = new MovieRepository();
$movieRepository->add(new Movie("God's Not Dead", 2014, 'Harold Cronk'));
$movies = $movieRepository->getMoviesByYear(2016);
$movieRepository->changeDirectorName('Mel Gibson', 'Gibson, Mel');

class Movie {
  public $_id = NULL;
  public $title;
  public $year;
  public $director;

  public function __construct($title = NULL, $year = NULL, $director = NULL) {
    $this->title = $title;
    $this->year = $year;
    $this->director = $director;
  }

  public static function from_array(array $arr): Movie {
    $instance = new Movie();
    $instance->_id = $arr['_id'] ?? NULL;
    $instance->title = $arr['title'] ?? NULL;
    $instance->year = $arr['year'] ?? NULL;
    $instance->director = $arr['director'] ?? NULL;
    return $instance;
  }

  public static function from_object(object $obj): Movie {
    return self::from_array((array)$obj);
  }
}

class MovieRepository {
  private $storage;
  public function __construct() {
    $this->storage = new JsonStorage('movies.json');
  }
  private function convert(array $arr): array {
    return array_map([Movie::class, 'from_object'], $arr);
  }
  public function all() {
    return $this->convert($this->storage->all());
  }
  public function add(Movie $movie): string {
    return $this->storage->insert($movie);
  }
  public function getMoviesByYear(int $year = NULL): array {
    return $this->convert($this->storage->filter(function ($movie) use ($year) {
      return $movie->year === $year;
    }));
  }
  public function changeDirectorName(string $old, string $new) {
    $this->storage->update(
      function ($movie) use ($old) {
        return $movie->director === $old;
      },
      function (&$movie) use ($new) {
        return $movie->director = $new;
      }
    );
  }
}

Takeaways

• The data is in the center
• Data structure driven file storage
• JsonStorage, Repository classes