Zotero - Obsidian Integration with Templates

Although all annotations are imported into Obsidian without templates, you will lose many metadata of your notes. Without a template, you are missing two types of metadata:

• Metadata about the Zotero bibliography item (author, title, year, publisher, location, abstract, item tags, etc.)
• Metadata about the annotation (colors and tags of the annotations)

You also lose control of where to store the files for notes and images (default: your attachment folder) and to store images as JPG (default: PNG).

Creating a Zotero Integration template is a complicated process for two reasons:

1. You have to know how to apply the Nunjucks template syntax. Nunjucks is a JavaScript templating language, supported by all modern browsers.
2. You must know the names of the variables you want to import. You can inspect the variable names with the “Data Explorer” command of the Obsidian-Zotero Integration plugin.

After choosing the “Data Explorer” command, Obsidian will open a new horizontal section with two buttons on the top.

1. Prompt for Selection: It displays a dialog window to choose your library and Zotero item you want to import. Please be patient — Obsidian may take seconds to open the dialog window.
2. Preview Import Format: This is the default value of this menu (button). It will display all the variable’s names and values for the Zotero item. However, you can also select one of your templates defined in the Zotero Integration plugin to see the result. The “Data Explorer” is an important tool for creating and improving Obsidian-Zotero Integration templates.

Template Examples

---
UID: <% tp.file.creation_date("YYYYMMDDHHmm") %>
created:
updated:
note: fleeting
status: to_process
language: english
source: 
category: 
subject: 
alias:
cssclass: literature-note
alias: [{% if shortTitle %}"{{shortTitle | safe}}"{% else %}"{{title | safe}}"{% endif %}]
---


{%- macro colorValueToName(color) -%}
    {%- switch color -%}
         {%- case "#ffd400" -%}
            Running Text (Yellow)
        {%- case "#ff6666" -%}
            Concepts, Definition, Important (Red)
        {%- case "#f19837" -%}
            ToDo (Orange)
        {%- case "#5fb236" -%}
            Resources (Articles / Books / Package / URLs)
        {%- case "#2ea8e5" -%}
            (R) Code (Blue)
        {%- case "#e56eee" -%}
            Chapter (Magenta)
        {%- case "#a28ae5" -%}
            Section (Purple)
        {%- case "#aaaaaa" -%}
            Figure, Table header, TOC Summaries (Gray)
        {%- default -%}
            No Color 
    {%- endswitch -%}
{%- endmacro -%}

{%- macro calloutHeader(type) -%}
    {%- switch type -%}
        {%- case "highlight" -%}
            Highlight
        {%- case "strike" -%}
            Strikethrough
        {%- case "underline" -%}
            Underline
        {%- case "image" -%}
            Image
        {%- default -%}
            Note
    {%- endswitch -%}
{%- endmacro %}

> [!info]
> - **Cite Key:** [[@{{citekey}}]]
{%- for attachment in attachments | filterby("path", "endswith", ".pdf") %}
> - **Link:** [{{attachment.title}}](file://{{attachment.path | replace(" ", "%20")}})
{%- endfor -%}
{%- if abstractNote %}
> - **Abstract:** {{abstractNote}}
{%- endif -%}
{%- if bibliography %}
> - **Bibliography:** {{bibliography}}
{%- endif %}
{%- if hashTags %}
> - **Tags:** {{hashTags}}
{%- endif %}


# Highlights by Sequence

## Imported on {{importDate | format("YYYY-MM-DD HH:mm")}}

{% for annot in annotations -%}
{%- if annot.annotatedText -%}
**Highlight**: {{annot.annotatedText | nl2br}}
{%- endif -%}
{%- if annot.imageRelativePath %}
![[{{annot.imageRelativePath}}]]
{%- endif %}
{%- if annot.ocrText %}
{{annot.ocrText}}
{%- endif %}
{%if annot.comment %}
**Comment**: {{annot.comment | nl2br}}
{%- endif %}
{% if annot.allTags %}
**Tag**: {{annot.allTags | nl2br}}
{% endif  %}
 [Page {{annot.page}}](zotero://open-pdf/library/items/{{annot.attachment.itemKey}}?page={{annot.page}}) {{annot.date | format("YYYY-MM-DD HH:mm")}}
***
{% endfor %}


{% persist "annotations" %}
{% set annots = annotations | filterby("date", "dateafter", lastImportDate) -%}
{% if annots.length > 0 %}


## Imported on {{importDate | format("YYYY-MM-DD HH:mm")}}
# Sorted by Colors
{% for color, annots in annots | groupby("color") -%}
#### {{colorValueToName(color)}}

{% for annot in annots -%}
> [!quote{% if annot.color %}|{{annot.color}}{% endif %}] {{calloutHeader(annot.type)}}
{%- if annot.annotatedText %}
> {{annot.annotatedText | nl2br}}
{%- endif -%}
{%- if annot.imageRelativePath %}
> ![[{{annot.imageRelativePath}}]]
{%- endif %}
{%- if annot.ocrText %}
> {{annot.ocrText}}
{%- endif %}
{%- if annot.comment %}
>
>> {{annot.comment | nl2br}}
{%- endif %}
>
> [Page {{annot.page}}](zotero://open-pdf/library/items/{{annot.attachment.itemKey}}?page={{annot.page}}) [[{{annot.date | format("YYYY-MM-DD#h:mm a")}}]]

{% endfor -%}
{% endfor -%}
{% endif %}
{% endpersist %}

/* 

Zotero Integration Plugin
Styling the template for Zotero Literature Notes

*/


/* Yellow */
.literature-note .callout[data-callout-metadata="#ffd400"] {
  --callout-color: 255,212,0;
}

/* Red */
.literature-note .callout[data-callout-metadata="#ff6666"] {
  --callout-color: 255,102,102;
}

/* Orange */
.literature-note .callout[data-callout-metadata="#f19837"] {
  --callout-color: 241,152,55;
}

/* Green */
.literature-note .callout[data-callout-metadata="#5fb236"] {
  --callout-color: 95,178,54;
}

/* Blue */
.literature-note .callout[data-callout-metadata="#2ea8e5"] {
  --callout-color: 46,168,229;
}

/* Magenta */
.literature-note .callout[data-callout-metadata="#e56eee"] {
  --callout-color: 229,110,238;
}

/* Purple */
.literature-note .callout[data-callout-metadata="#a28ae5"] {
  --callout-color: 191,127,191;
}

/* Grey */
.literature-note .callout[data-callout-metadata="#aaaaaa"] {
  --callout-color: 170,170,170;
}

Result of my template

My template consists of four main parts:

• 1. Metadata: This includes the YAML header, the abstract, and other important information (title, cite key, bibliography, tags).
• 2. Highlights by sequence: All annotations are sorted by their position in the document.
• 3. Highlights by color: Annotations sorted by their meaning. This requires a correspondence between color and purpose/reference type of the annotation.
• 4. Updates: Changes after the first import are added at the end of the Obsidian file. To get a file without updates, change the folder position of the file, rename the original file, or delete it.

I am planning to improve this template in two ways:

1. Only one admonition (callout) block for every color should exist.
2. Integrating the text of my green (H3) headers into the title of the admonition (callout) block.

Conclusion

With the annotation tutorial, I have demonstrated a modern technology supported annotation strategy. Every one of the three parts of the tutorial emphasizes one crucial aspect of this strategy:

1. Creating annotation with Zotero in PDFs, ePUB-eBooks or website snapshots.
2. Installing two plugins to build a bridge between Zotero and Obsidian to import notes created with Zotero from the annotated document.
3. Scraping annotations directly from the original document and redact them in a way to optimize further work in the note-taking app Obsidian.

Academic Workflow

In the first part of this annotation tutorial, I have described the different annotation types in Zotero version 7 and how to use them. In this second part, I will explain a simple method for importing Zotero annotations into Obsidian. These Obsidian notes will link to the source of the annotation (quote) in Zotero.

These backlinks are essential because one cannot always remember the original context of the citation. Especially if the creation of the annotation happened long ago or if the original purpose of the quote for a specific publication/research issue has changed, you need to check and reassess the context of the quote. Only by looking up the annotation in the source document can you prevent redundant and superfluous information from remembering the context of the quote.

After reading and annotating, it is crucial to elaborate on the highlighted content. The general strategy is to add, reformulate, and include the citations in the paper that is in progress. Whatever these changes are in the initial annotations, it is vital for an efficient workflow to always have a reference to the original position of the citation.

There are two strategies for the integration of your “research assistant” (Zotero) with your note-taking app (Obsidian):

1. Creating a note inside Zotero and importing it into Obsidian (this post).
2. Extracting all annotation directly from the document programmatically (the third part of the annotation tutorial).

In both cases, you need to install two plugins: Better BibTeX in Zotero and Zotero Integration in Obsidian.

Notes in Zotero

How to create a child note

Add Note from Annotations

1. Select the Zotero item or the document (PDF, ePUB-eBook, website snapshot) inside the Zotero item.
2. Open the context menu (right mouse click on my installation)
3. Select “Add Note from Annotations”

Add (empty) Note

1. Select the Zotero item (not the document).
2. Open the context menu (right mouse click on my installation)
3. Select “Add Note”

To repeat: An “child note” is a note that is sorted under a Zotero item, independently of where it was created initially.

We generate child notes as the easiest way to export notes into Obsidian. This has the disadvantage that changes in the Zotero annotations need another export to Obsidian. We will, therefore, cover in the third part of the tutorial an alternative, a more complex way where the Obsidian plugin extracts the annotations programmatically from the Zotero document.

Installing Plugins

Zotero Integration Settings

Introduction

The last task is to adapt the settings of the Obsidian Zotero Integration plugin.

1. Open Preferences with the menu “Obsidian → Preferences…” or CMD+, (macOS) / CTRL+, (Windows).
2. Scroll the left column down to “Zotero Integration” and select this plugin.

You will get a window with content similar to that in Figure 2. The look may differ depending on the Obsidian theme you are using. (I am using Cybertron)

The configuration setting page for the “Zotero Integration” plugin is complex. It consists of four parts:

• General Settings
• Citation Formats
• Import Formats (see third part of the annotation tutorial) and
• Import Image Settings

Although there are many preferences for the simple import method used in In this post, we will only need to complete the general settings.

Import Annotation into Obsidian without Template

For this procedure, you need to create the annotation file in Zotero manually, as I have already explained in Section 2.2.1.

1. Now change to Obsidian and choose either “Import notes” or “Insert notes into the current document” from the command palette. If you attach shortcuts to these procedures, you don’t need to call the command using the Obsidian palette.
2. You have to wait until a window with the Zotero database appears, where you can select the appropriate Zotero item. All note files inside the Zotero item will be exported. The first command (“Import notes”) creates a new file; the second command inserts the annotations into the cursor position of the current Obsidian document.
3. The results are annotations with two links: “Go to annotation” and the citation using the appropriate Zotero citation style. “Go to annotation” opens the PDF and jumps to the annotation’s location in the original document. The citation link selects the appropriate bibliography item in the Zotero middle pane.

Summary

In this post, I have explained how to integrate your “research assistant.” tool Zotero with the note-taking app Obsidian.

You can now elaborate on your notes using all the advanced features Obsidian offers!

In the third part of the annotation tutorial, I will explain how to use templates to programmatically extract annotations from Zotero documents. This has several advantages, but it is more complex to apply.

Are you satisfied with our result from this second part of the tutorial? Then, you can skip the last part or read it later when the need for better management of your different types of annotation arises.

Annotations in Zotero

I started using Zotero on December 30, 2007—yes, during the holidays! I always use the holidays to learn new apps and try things out. My Zotero database currently holds 5872 items created during more than 15 years of research. Until last year, I mainly used Zotero to produce correct formatted citations for publication. However, these changed when I learned new note-taking possibilities emerged with Zotero 6 (March 2022). It took me over a year to transform my workflow. Old habits die hard.

Book recommendation

For those of you that are interested to learn more about the philosophy and theory of annotation, I recommend the book-length treatment “Annotation” by Remi Kalir & Antero Garcia (2021).

There is also a website were you not only can you read the book for free, but you can also see how annotations features are used for an open peer review process of the book (Kalir and Garcia 2020).

Definition 1

Here, I will understand under the term “annotation” every addition anchored to the text and reinforces it. Annotations are a kind of paratext, produced outside the text but constantly referring and tied to it.

---

Eight types of annotations in Zotero

Zotero 7 has eight different types of annotations: highlighted text, underlined text (V7), “sticky” notes, adding text (V7), screenshots, drawing, resp. Ink annotation (V7), comments and tags.

Note 2: Annotations for PDF, EPUBs, and Webpage Snapshots

With Zotero version 7, the following explanations are not only valid for PDFs but also for eBooks in EPUB format and for snapshots of web pages!

1. Highlighting

To highlight text, click the symbol with the letter ‘A’ surrounded by a box at the top toolbar. You have eight colors for highlights at your disposal: yellow, red, green, blue, purple, magenta, orange, and gray. You can set your preferred color as the default color, which is, in my case, yellow.

2. Underlining (V7)

Note 3: Underlining as a new annotation type

Version 7 of Zotero supports underlining text with the eight colors mentioned in Figure 3.

Zotero treats highlighting and underlining as being very similar to highlights. Therefore, I refer to both annotation types whenever I talk about highlighting.

It is crucial to design a consistent system when to use each color. This will allow you to organize your annotations in Obsidian.

Some people have developed a system of argumentation types to distinguish the different highlighting colors. Below are two examples to give you a flair for the idea.

Color	mgmeyers	My own system
Yellow	Relevant / important	Running text
Red	Disagree	Important: Concept, Definition,
Orange	Question / confusion	ToDo
Green	Agree	Resources: article/book/package/URL
Blue	Relevant to current task	(R) Code; Example
Magenta	TODO / follow up	Section
Purple	Definitions / concepts	Chapter
Gray	Interesting but not relevant	Figure / table title

Both examples stem from times before Zotero 7. Therefore, there is no difference, and both annotation types will translate to the same meaning. I will explain more about the purpose of an appropriate classification system when we talk about Obsidian templates for the import of the Zotero annotations.

3. “Sticky” notes

To add a “sticky” note, click on the piece of paper symbol with the earmark. Then, you can click on a place in the PDF to generate the note. A “sticky” note symbol appears on the page, and on the left bar, a note box open for writing comments. You can move the “sticky” note around inside the page, but you can’t move it to another page (therefore “sticky”).

The primary purpose of “sticky” notes is to add a text annotation that relates to the text but not to a specific passage of the text. This can be, for instance, a comment on the structure of the text, a comment related to several paragraphs, a recurring argument, etc.

4. Text (V7)

With version 7, you can also write text directly into the document as an overlay. Zotero treats this as a comment, e.g., add tags or change the note text. But this annotation type is eye-catching, and you can move it around, but only on the same page.

5. Snapshots

By clicking on the black box surrounded by a rectangular selection, you can drag with the cursor to select the area of the screen to capture. You can add comments and tags as in the other annotation types, but you can also copy the picture or save it as a PNG file to your disk.

6. Drawings (V7)

Your pointing device (mouse, stylus, touchscreen) will be converted to a drawing device by clicking the ink button (with the drawing symbol). Selected the appropriate color, you can now create drawings overlaying the document.

7. Comments

You can add additional text to all types of annotations mentioned so far. Zotero will treat this additional text as your comment.

Warning 1: Changing highlighted text via the note?

Strange enough, You can also edit the highlighted text stored as annotation. Frankly, I do not know appropriate use cases because, with these changes, you would temper the original quote.

8. Tags

Another way to annotate a document is to add tags to highlighted text, “sticky” notes, screenshots, or comments.

You can select all annotations with the same tag inside the document (e.g., whenever the document is open and visible in the middle pane). Colors work as a tag, too. Hence, you can select all highlighted text in the same color. You can also combine different tags. For instance, two text tags with two colors will select all annotations that have one of these text tags or one of the chosen colors.

Outside the document, the tag works as a standard document tag in the tag pane at the bottom of the left column. Although the tags are attached to specific annotations outside the PDF (e.g., in the tag pane), they will select only the Zotero item with the document where the tag was used and not the appropriate annotations. But if you use the same tag for annotation in different documents, Zotero will open all items where these tags were applied.

We will see that after the export to Obsidian, these annotated tags gain power as the work across all Obsidian documents!

Summary

The Zotero documentation is not yet up-to-date, so detailed information is still lacking or incorrect. For instance, is The Zotero PDF Reader and Note Editor talking only about the PDF reader (version 6). The best page so far is the announcement page Zotero 7: Zotero, redesigned. However, many YouTube tutorials will explain the details of Zotero version 7 in several weeks.

In the meantime, the following annotated screenshot may be helpful. It recapitulates graphically the above textual description.

Research on Social Inequality

I plan to write blog articles about social inequality in Austria, my native country. To be innovative, I want to avoid repeating what other people (journalists, scientists) were saying (claiming) but to illustrate my thinking backed by original data. But where does the data come from?

Using the WID website

Replicating income distribution

Figure to replicate

As my test bed, I will use Figure 7 of BHE (Piketty 2022, 46).

Interactive Replication

As a first approach, I will replicate Figure 7 of BHE with the provided interactive tools of the WID. Here is a good starting point for income inequality.

From here, I will show screenshots for every step necessary to replicate Figure 7 of BHE, but here in this post, it is Figure 1. If you click on one of the small images below, it will open the figure in the bigger original size, and you can navigate back and forth between the figures in the table below.

Figures 3-6

---

---

---

Figures 7-10

---

---

---

Figures 11-14

---

---

---

Differences between original and replication

If we compare both figures side by side, we will see a significant difference. Our replication differs from the original in the years before 1900. For instance, in our figure, the people with the top 10% of income share about 60% of the total income, in contrast to the 50% share in the original graph. I need to find out where these differences come from. There may have been a correction of the computed extrapolations in an update after the book publication. In any case, this difference is not essential here. Remember, the primary purpose of the exercise is to understand the patterns of the database structure — and not (at least at the moment) to get correct figures.

There are other differences between the two pictures, but they are more of cosmetically nature:

• The choice of colors for the various time series is different.
• The labels of the X- and Y-axes are different. The original is labeled every 20 years and contains a vertical line every 10 years, whereas the interactive replication has a 25-year grid. The Y-axis in the replication is missing the %-sign.
• In the original, the time-series have different symbols in the data points (circle, triangle), which is vital in black & white (book) publications.
• The legend is in the original at a prominent place in the middle of the graph, whereas it is in the interactive replication at the bottom.

Display the results

Until now, I have used manually taken screenshots. But there are also several possibilities to display results offered by the WID.

Download a JPG-file

Note that there are now fewer labels on the X- and Y-axis. The legend’s appearance has changed somewhat, and some information that is irrelevant in a static view is removed.

Copy the URL

Another option is to copy the URL and integrate it as a link into the Quarto text. It opens up a web page with the figure and all the WID menus in your default browser. Others can explore the interactive graph or create their own figure from here.

See the example: Income inequality, France, 1820-2022

Embed as iframe

Another possibility is to embed the graph into an iframe.

Caution 1: Embedding WID iframes has some drawbacks

The legend overlaps the lower part of the graph. You have either to plan during the construction of the graph to prevent this drawback or (better) to adapt the code in the URL itself.

Besides, there is also an annoying cookies-consent message where the link to ‘learn more’ is wrong by pointing to another graph and not to the cookies policy.

The code in the above iframe is:

<iframe width="800" height="400" src="https://wid.world/share/#1/countrytimeseries/sptinc_p90p100_z;sptinc_p0p50_z;sptinc_p50p90_z/FR/2015/eu/k/p/yearly/s/true/9.4765/70/curve/true/1820/2022"></iframe>

It turned out that besides changing the delimiter for the axis from /9.4765/70/ to /0/62/ it is also necessary to enlarge the iframe’s height to position the legend lower. With trial & error experimenting, I got a good result after changing the height from 400 to 500. Here is the adapted iframe where you can see the full figure and also the slider to change the time period of the data:

Getting codes for indicators

By far, the most important result of our replication is that the URL contains the code for used indicators, areas, and percentiles.

https://wid.world/share/#1/countrytimeseries/sptinc_p90p100_z;sptinc_p0p50_z;sptinc_p50p90_z/FR/2015/eu/k/p/yearly/s/false/9.4765/70/curve/true/1820/2022

This is very helpful because it is now much easier to find the code explication in the Code Dictionary or the Summary Table. Let’s analyze the example of the sptinc_p90p100_z code:

• s: The first letter s determines the code series. s means “share” and contains a fraction between 0 and 1.
• ptinc: The next five letters are the code for the series concept. This is an essential help because a massive amount of five-letter codes exist. More than half of the 38 pages of the PDF version of the code dictionary lists five-letter codes and their meaning! Searching ptinc in the online Code Dictionary with the browser search tool results in “pretax national income”.
• p90p100: This one is easy. p90p100 is the top 10%, the percentiles characterizing people with an income equal to or higher than 90% of the population. Or p0p50 is the bottom 50% etc.

Additionally, we have the information from the legend that the units are adults, and the income is split equally between couples or other adult members (= 20 years or older) of the household. Therefore, we can add:

• 992: The age code description has three numbers. 992 stands for a population of individuals over age 20.
• j: The last letter is the unit description. j means equal-split adults.

With these data, we can now look for the corresponding codes in the database and filter the database to get only the required information. The variable name is, in our case:

{#eq-wid-variable-calculation}

If we look for this complete code in the Summary Table, we can read in the column “Simple description of the variable” the following explanation:

Pre-tax national income is the sum of all pre-tax personal income flows accruing to the owners of the production factors, labor and capital, before taking into account the operation of the tax/transfer system, but after taking into account the operation of pension system. The central difference between personal factor income and pre-tax income is the treatment of pensions, which are counted on a contribution basis by factor income and on a distribution basis by pre-tax income.

R Code

There is with {wid} a special R package for downloading data directly from WID (Blanchet 2024b). Read the instructions on the GitHub repo to get install information.

The R {wid} package should have a vignette with examples of its usage. But unfortunately, all the links to it are broken or not available except as .Rmd file in the GitHub repository itself. I have downloaded and rendered the vignette as a PDF for your convenience.

income_france <- wid::download_wid(
    indicators = "sptinc", # Shares of pre-tax national income
    areas = "FR", # France
    perc = c("p0p50", "p50p90", "p90p100"), # Bottom 50%, middle 40% and top 10%
    ages = 992, # individuals over age 20
    pop = "j" # equal-split adults
)

save_data_file("WID", income_france, "income_france.rds")

data <- paste0(here::here(), "/data/WID/income_france.rds")
income_france <- base::readRDS(data)
my_glance_data(income_france, seed = 123)

   obs country   variable percentile year  value
1    1      FR sptinc992j      p0p50 1820 0.1209
2   14      FR sptinc992j      p0p50 1923 0.1375
3  118      FR sptinc992j     p50p90 1910 0.3524
4  179      FR sptinc992j     p50p90 1975 0.4765
5  195      FR sptinc992j     p50p90 1991 0.4694
6  229      FR sptinc992j    p90p100 1880 0.5932
7  244      FR sptinc992j    p90p100 1927 0.4699
8  299      FR sptinc992j    p90p100 1982 0.2994
9  306      FR sptinc992j    p90p100 1989 0.3319
10 339      FR sptinc992j    p90p100 2022 0.3479

skimr::skim(income_france)

ggplot2::ggplot(income_france, 
    ggplot2::aes(x = year, y = value, color = percentile)) +
    ggplot2::geom_line() + 
    ggplot2::geom_point() +
    # ggplot2::geom_smooth(span = .9) + # for smoothing
    ggplot2::ylab("share of income") +
    ggplot2::scale_color_discrete(
        labels = c("p0p50" = "bottom 50%", "p90p100" = "top 10%", "p50p90" = "middle 40%")
    ) + 
    ggplot2::ggtitle("Pre-tax national income inequality in France, 1820-2022") +
    ggplot2::theme(legend.position = "bottom")

## save plot just once
# ggplot2::ggsave(filename = "BHE-sptinc992j_fr.png")

Comparison of my result with the original Figure 7

We are pretty near if we compare the above plots. Besides the mentioned difference in the magnitude of the top 10% and middle 40% before 1900, there are the following differences:

• title
• grid for x- and y-axis
• symbols for the data points
• position of the legend
• text of legend
• the smoothness of the time series

However, these differences are all solvable with the appropriate R code. I leave it here because the purpose of this article was to work with WID (and not to replicate figure 7 of BHE exactly.)

Conclusion

I have shown procedures for working with the inequality data of WID.

• One approach explores the interactive website on wid.world and tries to create the desired graph. There are several ways to use the resulting figure:
  • Downloading the picture as JPG
  • Embedding a link to the interactive graph on your website
  • Embedding the result as iframe in your website
  • Exploring the code in the URL and using it for the second approach (downloading the required data with R)
• The second approach uses the {wid} R package, especially the function wid::download_wid(). Knowing the required codes for the download function is essential. The first approach could help find the correct code composition in the WID code book, the Codes Dictionary or in the Summary Table.

Appendix

Introduction

Shared YAML metadata can be defined at both the project and directory level:

• project level: filename _quarto.yml
• directory level: filename _metadata.yml

For more information, see Project Basics – Quarto Shared Metadata

Project Level

On the project level, I will change only the provided URLs for my GitHub and Twitter accounts in this first step.

_quarto.yml

project:
  type: website

website:
  title: "Life Long Learning Lab"
  navbar:
    right:
      - about.qmd
      - icon: github
        href: https://github.com/
      - icon: twitter
        href: https://twitter.com
format:
  html:
    theme: cosmo
    css: styles.css

_quarto.yml

project:
  type: website

website:
  title: "Life Long Learning Lab"
  navbar:
    right:
      - about.qmd
      - icon: github
        href: https://github.com/petzi53/quarto-pb-blog
      - icon: twitter
        href: https://x.com/pbaumgartner
format:
  html:
    theme: cosmo
    css: styles.css

Directory Level

We have seen in an previous post that there is a _metadata.yml file in the posts folder.

_metadata.yml

# options specified here will apply to all posts in this folder

# freeze computational output
# (see https://quarto.org/docs/projects/code-execution.html#freeze)
freeze: true

# Enable banner style title blocks
title-block-banner: true

_metadata.yml

# options specified here will apply to all posts in this folder

# freeze computational output
# (see https://quarto.org/docs/projects/code-execution.html#freeze)
freeze: true

# Enable banner style title blocks
title-block-banner: false

# Author name of all blog posts
author: 'Peter Baumgartner'

# Table of content settings
toc: true
toc-depth: 4

page-layout: article

The changes are self-explanatory, so short comments will suffice:

• title-block-banner: I have already removed the title block banner from the home page listings and will disable it for blog posts as well. (I will revise this decision after I have written several posts. Maybe the title block banner is very useful for displaying a short description of the article.)
• author: All articles in my blog will be written by me.
• toc and toc-depth: I want a table of contents on the right side of the blog post, with a maximum of four levels. (I will revise this decision after I have written some longer articles. Maybe the TOC levels go too deep.)
• page-layout: I have already explained the page-layout: full directive from the title home page index.qmd. Now we want an article layout, because it creates a content area with a page-based grid layout that provides margins, areas for sidebars, and a reading width-optimized body region.

Git and GitHub

I will commit these third bulk of changes to Git and my demonstration repository quarto-pb-blog on GitHub.

Virgin Title Page

Here is the source code of the main index.qmd after the initialization:

index.qmd

---
title: "quarto-pb-blog"
listing:
  contents: posts
  sort: "date desc"
  type: default
  categories: true
  sort-ui: false
  filter-ui: false
page-layout: full
title-block-banner: true
---

And here is the look of the blog’s home page after the first rendering:

As you see the title banner shows twice the name of the R project folder ‘quarto-pb-blog’.

Change Title

The first line of the homepage’s YAML header in index.qmd is the title. If I change the title to “L3Lab,” only one part of the title banner will change.

The reason is that the upper left text is written in the Quarto project file.

_quarto.yml

project:
  type: website

website:
  title: "quarto-pb-blog"
  navbar:
    right:
      - about.qmd
      - icon: github
        href: https://github.com/
      - icon: twitter
        href: https://twitter.com
format:
  html:
    theme: cosmo
    css: styles.css

The top left name is the website’s title (and not the blog title). A website can have several components, such as a web page and a blog. I will call the website “Life Long Learning Lab,” the blog’s name remains “L3Lab”.

The website’s header doesn’t look nice. I will change the blog’s design (color, font size, layout, etc.) later in a separate post.

Explore Listing

Contents Directive

Listings enable you to automatically generate the contents of a page from a list of Quarto documents. In our case, the blog’s title is actually the home page’s listing title.

Instead of listing: default for a list of all documents in the directory and a drop-down menu for sorting direction, order criteria, and a search bar, we have specified with content: posts that only the documents in the folder “posts” should be listed. Have a look into the “posts” folder by selecting the “Files” tab in the bottom right pane of RStudio and click on the “posts” folder to open the directory.

The two folders “post-with-code” and “welcome” contain the two dummy articles generated automatically in the initialization process. I will explain the function of the _metadata.yml file later in another post.

Besides to list all articles of a folder, you can also write much more complex rules for including content by using globs and a list of targets in the contents directive. See for more details the two sections of the Quarto documentation: Listing Contents and Custom Listings.

Listing Types

There are three built-in types of listings you can choose from:

Type	Description
default	A blog-style list of items.
table	A table of listings.
grid	A grid of listing cards.

I am going to use for my blog the default blog-style list of items. However, it is instructive to see the other types to get a sense of their different appearances.

Grid Listings

Table Listings

Default Listings

Sort articles

Using the sort option controls the order of the listing. Each sort key can include a field name and optionally either asc or desc to control whether to sort in ascending or descending order. Since the default order is ascending, I had to add desc to get the most current article at the top. The sort key can also contain more than one field to sort.

Listing options

For every listing type, there are several listing options to specify details, such as:

• The maximum number of items: max-items
• The height of the image: image-height. The width is automatically determined and the image will fill the rectangle without scaling (i.e., cropped to fill).
• A placeholder for the image: image-placeholder. The default image for items if they do not refer to a special image. For more options, see Listing Options and Advanced Options.

Categories

Listings can also automatically add a list of categories to the page on which they appear. To turn on this behavior, you must set the option for categories to true: categories: true. You can choose between a few different category display styles: numbered, unnumbered, and word cloud.

When users click a category, the page will be updated to show only a listing of the items that match the selected category.

Sort- and Filter- Interface

Listings support interactive tools that allow the listing viewer to sort, filter, or page through listings.

I have turned off the sort and filter user interface and applied the default number of items per page for the default listing type: page-size: 25.

Page Layout

Quarto provides three different default layouts for HTML pages:

• The article layout provides a content area with a page-based grid layout that provides margins, areas for sidebars, and a reading width-optimized body region. I am going to use this layout type for article content. See for more information on article layout.
• The full layout uses the article grid system but automatically expands the content area to use the sidebar and margin region if no content is placed within those regions. This is useful for layouts that don’t need to be constrained to reading width, and that will benefit from additional horizontal space (e.g., landing or index pages). It is also often appropriate for listing pages.
• The custom layout provides the possibility to create an individual layout. Learn more on the Page Layout page.

Title Block Banner

HTML pages rendered with Quarto include a formatted title block at the start of the article. The title block contains the title, subtitle, authors, date, doi, and abstract. There are several options to style the title block. In addition, if you’d like a more prominent title block, you can use title-block-banner to create a banner-style title block.

For my blog, I don’t want (at the moment) a title block banner. Setting title-block-banner to false saves space on the title page and feels more prosaic and less business-like.

Git and GitHub

Here I will pause and commit these changes to Git and GitHub. It is interesting to note that there are – besides index.qmd and _quatro.yml – several other files that have changed.

I haven’t even touched many of these files. But the title change propagates the new name into all blog post titles of articles already written and also in the corresponding navigation bars. This is shown with the example of the about page in the next screenshot (Figure 10).

And I have committed all these changes I’ve got with this second commit a new status in my GitHub repo.

Summary

I have explained the different YAML directives for the blog’s homepage file index.qmd, which are generated automatically when the Quarto blog is initialized. During this tour I have changed two options:

• The title of the blog listing in index.qmd and of the website in _quarto.yml.
• I got rid of the title block banner.

To follow the next sequence of my blog tutorial, check if you have the same starting position.

The changes of the source code of the home page index.qmd result to:

index.qmd

---
title: "L3Lab"
listing:
  contents: posts
  sort: "date desc"
  type: default
  categories: true
  sort-ui: false
  filter-ui: false
page-layout: full
title-block-banner: false
---

There are also a change in the website title in the _quatro.yml file:

_qarto.yml

project:
  type: website

website:
  title: "Life Long Learning Lab"
  navbar:
    right:
      - about.qmd
      - icon: github
        href: https://github.com/
      - icon: twitter
        href: https://twitter.com
format:
  html:
    theme: cosmo
    css: styles.css

Then I have committed these changes to Git and my dummy GitHub repo.

Why using Git and GitHub?

After initializing a new Quarto blog I will create the initial (first) Git commit, initialize a GitHub repo, and push the content of the initial commit into the new repository.

There are several reasons to use Git:

• A Git repository (or repo) holds all the project files and their entire revision history. It is therefore not only easier to track changes but also to restore a previous version if necessary. (See the nice introduction)
• GitHub is an online platform to store and manage your content (code but also text and other resources), as well as track changes to it. It facilitates collaborations with your team but also with other (unknown) people.

Procedure for using Git

In the previous blog post, we have already initialized the Quarto weblog, including Git support, by checking the appropriate check box.

1. To see all the changes you have made so far, click on the Git tab at the top right pane of RStudio. You should see a picture similar to the following one.

2. After selecting all files and checking them (checking a single file of a group of selected files checks all selected files), the Git window shows all files seen above but also the files inside folders that are now prepared for the next Git commit.

3. After clicking on the “Commit” tab (or using the shortcut CTRL-ALT-M) a window opens for writing the commit message. Write short notes using a commando language. For example a commit text for the main index.qmd file could be: “change title-block-banner to true”.
   
   It is usual to start the commit messages with “initial commit”. I had erroneously used “first commit” in the screenshot.

4. After clicking the Commit button, RStudio presents a window with the results of the action:

Closing both windows will end the Git procedure to the local repository.

Procedure for initializing a GitHub repo

Next create a GitHub repo with all the locally committed content. I am using here the {usethis} package because it automates all the necessary steps. However, — as I mentioned above — there are several requirements before you can create and connect a GitHub repo. With Happy Git and GitHub for the useR you should read and learn how to connect an RStudio project to GitHub under different conditions. In this case, I have applied Chapter 17, “Existing project, GitHub last”.

1. Write usethis::use_github() into the console window of the RStudio window (lower left pane). After return your console will display the following lines:

The screenshot above shows the different actions for the usethis::use_github() command. In the last step, the newly generated and updated repository on GitHub was opened.

Summary

This post shows how to initialize Git and GitHub using RStudio after crating of the brand-new Quarto blog:

1. Commit all the files generated automatically after launching the Quarto blog (as described in Initializing a Quarto Blog with RStudio) to the local Git repository.
2. Initializing a remote GitHub repository, connecting the local repo with it, and connecting it with the local repo.

Initializing a Quarto Blog

There are several ways to start a new Quarto blog. I will use a menu driven approach using RStudio. For the following workflow you should have installed RStudio and have some knowledge how to use it.

I will not explain everything related to creating a new Quarto blog; instead, I will focus on my choices to initialize the blog.

I am using

• R version 4.4.1 (2024-06-14)
• Quarto version 1.5.54

New Project…

Inside RStudio, use the New Project… command and select New Directory and then Quarto Blog.

• I am using {knitr} as the engine.
• The {renv} package is for creating reproducible environments for R projects. I will not use it as I have no experience with this complex package, and reproducibility is not an issue for my private blog (at least, I think so at the moment).
• Selecting “Use visual markdown editor” is not important, as this option is for all .qmd files in RStudio always available. I think that this option only controls the default behavior whenever a new file for a post is created.

Key Files

Clicking the “Create Project” button generates different key files and opens the _quarto.yml file in the RStudio editor window.

If you can’t see all five files starting with a dot (so-called dotfiles) in your installation, go to the drop-down menu ‘More’ and check the last option ‘Show Hidden Files’.

Here’s a summary of the key files created within the starter blog project:

File	Description
_quarto.yml	Quarto project file.
index.qmd	Blog home page.
about.qmd	Blog about page.
posts/	Directory containing posts
posts/_metadata.yml	Shared options for posts
styles.css	Custom CSS for website

_quarto.yml

project:
  type: website

website:
  title: "quarto-pb-blog"
  navbar:
    right:
      - about.qmd
      - icon: github
        href: https://github.com/
      - icon: twitter
        href: https://twitter.com
format:
  html:
    theme: cosmo
    css: styles.css

index.qmd

title: "quarto-pb-blog"
listing:
  contents: posts
  sort: "date desc"
  type: default
  categories: true
  sort-ui: false
  filter-ui: false
page-layout: full
title-block-banner: true
---

Render Blog

You can control via Tools->Global Options…->R Markdown->Show output preview in… whether the rendered result appears in the browser window or in the viewer pane. Generally, I use the “Window” option because it gives me a better view of the real setting.

RStudio console output

==> quarto preview index.qmd --to html --no-watch-inputs --no-browse

pandoc to: html
  output-file: index.html
  standalone: true
  title-prefix: quarto-pb-blog
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  css:
    - styles.css
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  theme: cosmo
  title: quarto-pb-blog
  listing:
    contents: posts
    sort: date desc
    type: default
    categories: true
    sort-ui: false
    filter-ui: false
  page-layout: full
  title-block-banner: true
  
WARN: Unable to read listing item description from posts/post-with-code/index.html
WARN: Unable to read listing item description from posts/welcome/index.html
WARN: Unable to read listing preview image from posts/welcome/index.html
Output created: _site/index.html

Preparing to preview
[1/4] posts/post-with-code/index.qmd
R version 4.4.1 (2024-06-14)

processing file: index.qmd
                                                                                                            
output file: index.knit.md

[2/4] posts/welcome/index.qmd
[3/4] about.qmd
[4/4] index.qmd

Watching files for changes
Browse at http://localhost:5146/
GET: /

These warnings are prompted by the fact that we need to render the complete blog with SHIFT-CMD-B or by clicking “Render Website” after choosing the “Build” button on the upper right RStudio pane. I will explain render strategies in a later post.

Summary

This is the end of the initialization process. In later blog posts I will add more functionality and style the web page it to get a more attractive appearance of the Quarto blog.

Motivations for writing blog posts

My conception, written in my first blog entry, needed to be revised. The problem was a certain kind of perfectionism: I thought that my knowledge about R issues was not deep enough to be interesting to other people. After visiting the useR!2024 conference, I changed my view in four aspects:

• R has a dynamic community where new people are constantly attracted. My meager knowledge could be helpful, especially for beginners, as I was struggling with the same starter issues not long ago. Sometimes, a person with intermediate knowledge can be more beneficial than an expert.
• The R project is a very complex field. Nobody has the same level of expert knowledge in every area. Therefore, it is possible that some parts of my knowledge could be useful for other people, even if they are (in other areas or generally) much more advanced.
• Putting my understanding into words so that other people can use this information for their project is an important exercise: It ensures that I have grasped the essentials and fosters my learning by writing down all the necessary details.
• The most important reason to write blog posts about issues I have learned is the “Future me.” After several months of working on other topics, I often notice that I have forgotten some details of procedures I have already learned.

To elaborate on the last point: In one case, I was looking for a solution to a problem I experienced. After several web searches, I found the solution on StackOverflow. I was shocked when I noticed that it was me who wrote the correct answer several years ago!

A new start

After the useR!2024, I was very motivated to resume the Quarto blog I started in April 2023. I noticed that there are two versions: One published via Netlify and another one – already pretty elaborated – locally.

_quarto.yml

    right:
      - text: Proudly powered by <a href="https://quarto.org/">Quarto</a>

_quarto.yml

    right:
      - text: Proudly powered by <a href=https://quarto.org/>Quarto</a>

Help page about article layout

When I looked at the Quarto help pages, I found a chapter about article layouts. But it treated the layout of pages in a more general way by answering the following questions:

• How to use the whole main content region?
• How can content overflow this standard content region?
• How can one span the width of the entire page?
• How can authors place content on the document margin?

I think that these questions address pretty special requirements. I just wanted a solution for content items side by side. This support page about the article layout was not helpful to me.

Using a table failed

My first approach was to use a table to position images side by side. I created a one-row / two-column table with the visual editor of RStudio and uploaded in each column a picture. But the screenshots didn’t appear. Whenever I rendered the page, Quarto told me, “404: Not Found”.

As I inspected the source code, I saw the reason: The long image URLs didn’t fit in the table column, and therefore forced line breaks that destroyed the URLs.

Using pandoc extensions

After long research, I found three answers to my question via StackOverflow. They all used the fenced_divs extension of pandoc.

It is easier to explain the method after I have presented these three examples. I will use the same toy text sample for the following illustration.

Text in the left column:

I would like to have text here

Sentence becomes longer, it should automatically stay in their column

Text in the right column:

and here

More text

---
title: "Two columns separated by a small third column"
format: html
engine: knitr
---

:::: {.columns}

::: {.column width="30%"}
I would like to have text here

Sentence becomes longer, it should automatically stay in their column
:::

::: {.column width="10%"}
<!-- empty column to create gap -->
:::

::: {.column width="20%"}
and here

More text
:::

::::

---
title: "Two columns using the bootstrap CSS grid system"
format: html
engine: knitr
---
  
  
:::: {.grid}

::: {.g-col-4}
##### First column 
I would like to have text here

Sentence becomes longer, it should automatically stay in their column
:::
  
::: {.g-col-4}
##### Second column 

and here

More text
:::
  
::::

---
title: "Two columns with text center-aligned"
format: html
engine: knitr
---

::: {.cell}
<style type="text/css">
.column {
text-align: center;
}
</style>
:::

:::: {.columns}

::: {.column width="30%"}
##### First column

I would like to have text here

Sentence becomes longer, it should automatically stay in their column
:::

::: {.column width="30%"}
##### Second column

and here

More text
:::

::::

Fenced divs in Pandoc: How do they work?

rmarkdown::find_pandoc()

$version
[1] '3.2.1'

$dir
[1] "/usr/local/bin"

Terminal (slightly redacted)

pandoc --version
(base) path@my_machine quarto-blog % pandoc --version
pandoc 3.1.2
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: /Users/path@my_direcotry/.pandoc
Copyright (C) 2006-2023 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.
(base) path@my_machine quarto-blog % 

A genuine Quarto solution

After I looked around at how to create a two-text column layout for a while, I found a simple Quarto answer. Hidden in the help page about figures, I found the remark:

You can treat any markdown content you want as a figure by enclosing it in Pandoc div block with an identifier prefaced with #fig-. See the first sentence under the header Figure Divs.

The first part of the sentence, “You can treat any markdown content you want as a figure…” could be reversed to the concept that a layout for figures works similarly to a design for text.

I came up with the following solution that I also published to answer the StackOverflow post.

---
title: "Two columns layout Quarto with equally wide columns"
format: html
engine: knitr
---

::: {layout-ncol=2}


First column

Second column

I would like to have text here

and here

Sentence becomes longer, it should automatically stay in their column

More text

:::

---
title: "Two columns layout Quarto with different wide columns"
format: html
engine: knitr
---

::: {layout="[[10,5], [40,20], [26,13], [2,1]]"}


**First column**

**Second column**

I would like to have text here

and here

Sentence becomes longer, it should automatically stay in their column

More text

:::

Conclusion

There are many possibilities for a creative layout design in Quarto. This article shows the use of Pandocs div block directives with five different methods. These layout directives can be generalized from a two-column to a many column layout

1. Several columns with varying percentages of widths.
2. Several columns using the very flexible bootstrap CSS grid system.
3. Several columns with formatted explicitly with a CSS style.
4. Several columns with the same width using the Quarto layout-ncol or layout-nrow attribute.
5. Several columns with different widths using the Quarto general layout attribute.

… deletion of the remote branch with GitHub is not enough

This situation happened to me after I proposed fixing a typo for a repo that had been frozen because of a necessary general overhaul. I thought to delete the submitted remote branch at the GitHub site would bring me back to my initial position.

But back in RStudio, it turned out that the deleted branch was still there.

Ok, I understood that my remote action did not delete the local branch too. But why was the remote branch still there? After all, I had it just removed. (I double-checked the deletion, and the remote branch at the remote GitHub site was, in fact, completely gone.

There are three, not two branches

After some research, I found an appropriate question-answer dialog (How do I delete a Git branch locally and remotely?) at StackOverflow with more than 11 million views! It seems that I am not the only one surprised by the situation.

After reading this and other answers at SO, I learned that there are three different branches:

When you’re dealing with deleting branches both locally and remotely, keep in mind that there are three different branches involved:

• The local branch X.
• The remote origin branch X.
• The local remote-tracking branch origin/X that tracks the remote branch X. Quote from the top second answer

I deleted the remote origin branch X manually at the GitHub site. The local branch but also the local remote-tracking branch were still functional!

For every one of these three branches, one needs different commands to delete them. And here – as far as I understood – we have met the limits of RStudio and/or {usethis}. For the following commands, you have to change from the console to the terminal.

Useful commands for deleting branches

Terminal

setopt interactivecomments
git push origin -d <branch>

Terminal

git checkout master # change branch, you can't delete a branch you are working on
git branch -d <branch>
git branch -D <branch> # force-delete un-merged branches

Terminal

git branch -dr <remote>/<branch> # 'dr' stands for "delete remotes"
git branch -dr origin/dev        # example

# Delete multiple obsolete remote-tracking branches
git fetch <remote> -p  # 'p' stands for "prune"
git fetch origin -p    # example

My first blog entry with Quarto

This is my first entry in a new blog powered by Quarto. After continuously struggling with the complex machinery blogdown - Hugo - Wowchemy, I feel like coming home from a trip abroad!

Following the Quarto guide Creating a Blog, it took me only 2 (!) minutes to render the provided blog template locally. After just one hour, I had my blog using Quarto, GitHub, and Netlify online. And half of this time, I spent thinking about a name for my blog and what to do with my older and outdated other blogdown web presences. (To put this quick access into perspective: I already had a GitHub account, GitHub credential, and a Netlify account. And I didn’t work on the content of a blog entry but used just the installed test pages.)

I never was happy with the fast development of Hugo that brought with it many changes and a considerable learning overhead. The same problem happened with the Academic theme by Wowchemy. The end product looked nice and ran fast, but I had already forgotten some of the many details for a good-looking blog post after two weeks. As a result, I spent more time learning the blog machinery than I had time to write about substantive subjects.

It was helpful when Yihui Xie introduced `blogdown::checksite()` and the possibility of freezing to a specific Hugo version. But the work in the blogdown-Hugo environment was never intuitive and satisfying. I think it’s self-explanatory that the introductory message to the blogdown book for over two three years now states as a note from the authors: “Some of the information and instructions in this book are now out of date because of changes to Hugo and the blogdown package.”

Future plans

I am sure there will also be some intricacies I must learn and overcome with Quarto, but this will conform with all my other R knowledge. At the moment, my feeling is enthusiastic: It is easy and intuitive to write on this platform. I hope to overcome technical barriers and concentrate on thematic topics finally. Since English is not my mother tongue, it is already difficult enough to express myself precisely about complex content-related questions. I don’t need any additional technical difficulty in writing blog posts.

In contrast to my previous website projects, I am planning two different strategies: 

• I will not go into the details of the website design immediately but will work on these issues from time to time.

• I will not announce the website until I am sure I will use it regularly and long-term. I think that about 10 articles – written in two months – should give me the trust that I will use it sustainably.