workshops

Georeferencing the Internet Archive with Allmaps

Thursday, December 07, 2023, 5:00 PM - 5:30 PM CET

Jules Schoonman & Bert Spaan

2023 Online Meeting, IIIF Consortium

Abstract

Following the exciting news about the collaboration between the IA and IIIF Consortium, this workshop explores the possibilities of annotating IA material using the Allmaps platform. We will locate relevant collections and resources in the IA, and open them in the Allmaps Editor using the available IIIF Manifest URIs. We’ll look at the format of a Georeference Annotation and demonstrate its usages in the Allmaps Viewer and other tools. When working with open data that is not yet available elsewhere with IIIF, the IA can thus serve as a repository to launch geospatial research projects. This workshop will give you everything you need to get started–and the approach can be replicated for other IIIF content as well. When we run into problems, we’ll create issues in the relevant repositories to help the development of both Allmaps and IA’s IIIF implementation.

Step 1: Finding Maps in the Internet Archive

As of September 2023 the IIIF API of the Internet Archive was upgraded in collaboration with the IIIF Consortium. The implementation is still under development and some items might throw an error when opening them in Allmaps. Please notify us when this happens and we’ll make sure to investigate and publish a fix when possible (or raise an issue in the IIIF repository of the IA).

Some collections of interest:

• Center for Sacramento History
• United States Geological Survey
• Getty Research Institute
• Smithsonian Libraries and Archives
• Digital Piranesi
• David Rumsey Map Collection (Please note that the images on the David Rumsey website have a higher resolution than those in the Internet Archive and also support IIIF.)

How to find the IIIF Manifest of an item in the Internet Archive?

• Locate the id in the URL: https://archive.org/details/{id}/
  • In the URL https://archive.org/details/map2002023007/ the id is map2002023007
• Add the id to the following pattern: https://iiif.archive.org/iiif/{id}/manifest.json
  • E.g. https://iiif.archive.org/iiif/map2002023007/manifest.json

💡You can use this bookmarklet to quickly open the IIIF Manifest of an item in the Internet Archive and this bookmarklet to open the current browser address in the Allmaps Editor. Use those two in sequence. Alternatively, you can use this form on the Allmaps documentation site (which is still in development).

❗Allmaps does not currently support IIIF Collection Manifests

Step 2: Georeferencing Maps in the Allmaps Editor

• Copy the URL of the IIIF manifest
• Open the Allmaps Editor
• Paste the link in the field and click Load. At the top you now see four tabs (you can always go back and forth). If you get an error message, double check if you copied the right link.
• Select an image under the Collection tab (often there’s only a single image to select). The two icons indicate whether a map already contains a mask and/or control points.
• Click the Mask tab and draw a mask by zooming in and clicking the image. The mask will be used to crop the map in the viewer. If you want to start again, click the middle button of the menu on the bottom right. This opens up a panel listing all masks; click the bin to remove one (there’s no way to undo this currently).
• You can move points around or add new points to the mask by dragging points or lines. You can also remove a point by clicking it while holding shift.

💡If you accidentally create a second mask, you can quickly change tabs (to Collection and back) to remove it again.

• Now go to the Georeference tab. On the left you see the selected image, on the right a world map. Zoom to the relevant region of the map on the right. Click a point you recognise on both maps (which has remained the same over time), such as a church tower or a road crossing. Add the point on both sides (a number will indicate that they are paired). Add at least three of these points (they are called control points) and try to spread them as much as possible. If you unfold the layers panel (middle button of the menu) you obtain an overview of your mask and the control points. You can remove points if needed (there’s no undo function yet). You can also select another mask and add points for that selection.

❗Removing points from the menu might confuse the numbering. Reload the page to reset the numbering. Note that the order of numbers change after reloading the page.

💡You can rotate the view by holding Option/Alt-Shift.

💡You can load a custom reference map by following the instructions in this Observable Notebook.

Step 3: What is a Georeference Annotation?

• After finishing Step 2, Open the </> panel on the bottom right.
• Next to Show annotation for: you can choose between Manifest, Image, Map.
  • Map refers to the current selected mask (and related control points)
  • Image refers to all the masks (and related control points) of the currently selected image (or canvas)
  • Manifest refers to all the masks (and related control points) of all the images in the current IIIF Manifest (see the overview in the Collection tab).
• Select Map and click Open in new tab. You now see a json document that (almost 😅) conforms to the specifications of the Georeference Extension. We’ll explain the different parts to make you familiar with the structure.

❗The Allmaps Editor currently stores the annotations for you. If you open the same map, the masks and control points are preserved. If someone else opens the same map, they can alter the annotation. It is therefore wise to save a copy of your data on your hard drive or elsewhere. The Editor does not yet support loading custom annotations, but the Allmaps Viewer does.

Step 4: Opening a Georeference Annotation in the Allmaps Viewer

• Go back to the Editor and click Copy in the </> panel.
• Navigate (in a new tab) to the Allmaps Viewer, paste the Georeference Annotation in the second field and click View. The georeferenced map will now open in the Allmaps Viewer. You can go back and forth between the original image and warped map by clicking the buttons in the top right corner. If the IIIF manifest contains multiple images and if more than one were georeferenced (and exported), all images are rendered at once. Some tricks:
  • Hold space to quickly hide the map
  • Press B or use the right knob to remove the background color
  • Press M to display the outline of the mask(s)
  • Press T to change the transformation algorithm
  • In case of multiple maps use [ and ] to browse the collection (the selected map will be on top)
  • Right click on a map to change the layer order

💡 The copy/paste method serves to demonstrate how Allmaps works on the basis of Georeference Annotations. An easier way is to navigate to the Results tab and open one of the links. In this case, the annotation is loaded from Allmaps’ database directly. This means that new edits in the Editor will become visible in the Viewer after refreshing the page. You can leave both tabs open to improve your work.

💡 You can also use the following method to see if an annotation exists of a certain manifest:

• Copy the IIIF manifest link
• Open this URL: https://annotations.allmaps.org/?url=
• Paste the IIIF manifest link after ?url= in the browser address bar and press enter

(There’s also a bookmarklet for that.)

Or you can load a Manifest URL directly in the Allmaps Viewer.

❗Allmaps references Manifests by using a hash of the Manifest ID property. This can be different from the URL. In the case of the Internet Archive, the ID of https://iiif.archive.org/iiif/3/postofficemanual00unit/manifest.json is https://iiif.archive.org/iiif/postofficemanual00unit/manifest.json (without the 3/ in the URL). More about IDs in Allmaps

Step 5 (extra): Tracing maps with geojson.io

• In order to add the map as a layer in geojson.io (and also in GIS-applications), you need to use the Allmaps Tile Server. The Tile Server acts as a proxy server to translate IIIF images to XYZ map tiles, using the georeferencing annotation.
• There’re three ways to generate the URL template for XYZ map tiles:
  • Copy the XYZ link at the bottom of the Results tab of the Allmaps Editor.
  • Follow the Tile Server instructions and click the Copy URL template to clipboard button
  • Copy the URL to the georeferencing annotation (see previous section) and place it after = in the following URL: https://allmaps.xyz/{z}/{x}/{y}.png?url=
• Go to the geojson editor
• Click Meta and Add raster tile layer in the top menu. Paste the URL template and give the layer a name. Navigate the map to see the result (it won’t do this automatically). At the bottom right, a checkbox appears to hide/show the layer. (You can repeat these steps to add another layer.)
• In the geojson.io editor you can add markers, rectangles, polygons and polylines (see the tool palette). You can also edit and delete features by using the buttons at the bottom of the tool palette (follow the on-screen instructions).
• When you click a feature, you can edit its metadata in a popup panel. Each row in the table represents a label-value pair: labels left and values right. You can add a new pair by adding a new row.
• Don’t forget to save the data!

❗You can also load XYZ maps in Placemark. This app supports the TileJSON format which can be requested by using the following pattern: https://allmaps.xyz/tiles.json?url=.