v2.0.2

• Fix migration order.

• Don't display the expected "Memstore.Init: json: cannot unmarshal number / into Go struct field storedSession.paths of type int64" error log on startup; this got displayed once, but was a bit confusing.

• Display a message on startup after the first update to direct people towards the 2.0 release notes and "goatcounter reindex".

Be sure to read the release notes for 2.0.0 as there are some incompatibilities and manual actions!

v2.0

Version 2.0.2

v2.0.1

• Fix migrations 🤦 They worked when they were written, but a bunch of things changed in GoatCounter and some older ones didn't run any more.

• Add -test flag to goatcounter db migrate to rollback a migration, so it's easier to test if migrations will run correctly without actually changing the database.

Be sure to read the release notes for 2.0.0 as there are some incompatibilities and manual actions!

v2.0.0

The version is bumped to 2.0 because this contains a number of incompatible changes: several CLI commands got changed, and it includes some large database migrations – running them is a bit more complex than the standard migrations.

An overview of incompatible changes:

• There are some rather large changes to the database layout for better efficiency; this means:

  • Somewhat faster queries.
  • Greatly reduced disk space requirements for the database.
  • The Browsers, systems, size, and location stats are filtered if you enter something in "filter paths". Previously this always displayed the site totals.
  • "Purge path" now works as expected for all stats.
  • Easier to add new statistics in the future.

  To update:

  1. You must first update to 1.4.2 and run all migrations from that. Updating from older versions directly to 2.0.0 will not work!

  2. Run the migrations with goatcounter serve -automigrate or goatcounter migrate.

  3. You probably want to manually run VACUUM (or VACUUM FULL for PostgreSQL) after the migration to free up unused rows. This isn't strictly required, but frees up disk space, and removes some of the autovacuum pressure that will run in the background.

  4. Run goatcounter reindex.

  All of this may take a while if you've got a lot of data. For about 500,000 pageviews it takes about 3 minutes on SQLite, but if you've got millions of pageviews it may take an hour or more.

  If you want to keep pageviews while this is running you can:

  1. Write it to a logfile from a proxy or temporary HTTP server and run goatcounter import on this after the migrations are done.

  2. Use goatcounter buffer.

• goatcounter migrate is now goatcounter db migrate. It also behaves a bit different:

  • goatcounter db migrate pending lists only pending migrations, and will use exit code 1 if there are any pending migrations.
  • goatcounter db migrate list lists all migrations, always exits with 0.

• If you use PostgreSQL you need PostgreSQL 12 or newer; this was already the case before and you could run in to some edge cases where things didn't work, but this is enforced now.

• The none value got removed from the -tls flag; use tls=http to not serve TLS. This was confusingly named as you can do -tls=none,acme to still generate ACME certificates, but none implies that nothing is done.

• goatcounter create is now goatcounter db site create, and some flags got changed:

  • -domain is now -vhost.
  • -parent is now -link.
  • -email is now -user.email.
  • -password is now -user.password.

• The -port flag for goatcounter serve is renamed to -public-port. This should clarify that this isn't the listen port, but just the port GoatCounter is publicly accessible on.

• The -site flag got removed from goatcounter import; you can now only use -url to set a GoatCounter site to import to. The automagic API key creation was more confusing than anything else.

  You can use goatcounter db create apitoken to create an API key from the CLI.

• If you build from source, the build flag to set the version changed from:

  -ldflags="-X main.version=..."
  

  to:

  -ldflags="-X zgo.at/goatcounter.Version=..."
  

• The CSV export format was increased to 2; it now includes the parsed browser and system values in addition to the User-Agent header. Version 2.0 will not be able to import the older exports from version 1.

Other changes:

• You can read pageviews from logfiles with the goatcounter import command; you can also send pageviews to goatcounter.com with this (you don’t need to self-host it). See goatcounter help import and the site code documentation for details.

• You can now create multiple users; before there was always a single one. You can add users in Settings → Users.

  As a consequence, "Site settings" and "User preferences" are now split in to two screens. The Settings button in the top-right now displays only site settings, and clicking on your email address in the top right displays user preferences, which every user can configure to their liking.

• You can now configure what's displayed on the dashboard, in what order, and configure some aspects of various "widgets". You can set it in User preferences → Dashboard. Some settings from the main settings page have moved there.

• You can save a default view for the dashboard. Instead of always loading the last week by default, you can now configure it to load the last month, or view by day, or anything you want really.

• You can choose which data to collect; you can disable collecting any User-Agent, location, Referrer information.

• Ability to record state/province/district in addition to country, so it records "US-TX" or "NL-NB" instead of "United States" or "Netherlands".

  This option can be disabled separately from recording the country (enabled by default) and you can set which countries to record it for (defaults to US, RU, CH).

  This requires specifying the path to a GeoIP City database, which isn't included since it's ~30M.

• There are now stable count.v*.js scripts that can use subresource integrity. See the integration code for a list and hashes.

• You can use data-goatcounter-settings on the <script> tag to load the settings (requires count.v2.js or newer).

• New goatcounter buffer command; this allows buffering of pageviews in case the backend is down, running migrations, etc. See goatcounter help buffer for more information.

• The database for PostgreSQL is now created automatically; you no longer need to do this manually.

• You can copy settings from a site to other sites in Settings → Sites.

• Add goatcounter db command; you can now edit and delete sites, users, and API keys from the CLI. The create and migrate commands are now merged in to this as subcommands.

• Add a gcbench utility for inserting random pageviews in a database; for testing and comparing performance. This might be useful for end-users too in some cases, for example to see how much performance difference SQLite and PostgreSQL will give you, or to test if frobbing with server settings makes a difference:

  $ go run ./cmd/gcbench -db sqlite://db/gcbench.sqlite3 -ndays=90 -npaths=100 -nhits=1_000_000
  $ go run ./cmd/gcbench -db postgresql://dbname=gcbench -ndays=90 -npaths=100 -nhits=1_000_000
  

  Right now it doesn't try super-hard to simulate read-world usage patterns: the distribution is always uniform, but it still gives a reasonably accurate indication for comparison purposes.

• Many other minor changes and improvements.

• For changes since RC1 see: v2.0.0-rc1...v2.0.0

v2.0.0-rc1

The version is bumped to 2.0 because this contains a number of incompatible changes: several CLI commands got changed, and it includes some large database migrations – running them is a bit more complex than the standard migrations.

Because this includes so many changes I'll do a Release Candidate first. Please report issues if you have any problems! Barring unexpected large issues I'll do a 2.0 release next week or so.

An overview of incompatible changes:

• There are some rather large changes to the database layout for better efficiency; this means:

  • Somewhat faster queries.
  • Greatly reduced disk space requirements for the database.
  • The Browsers, systems, size, and location stats are filtered if you enter something in "filter paths". Previously this always displayed the site totals.
  • "Purge path" now works as expected for all stats.
  • Easier to add new statistics in the future.

  To update:

  1. You must first update to 1.4.2 and run all migrations from that. Updating from older versions directly to 2.0.0 will not work!

  2. Run the migrations with goatcounter serve -automigrate or goatcounter migrate.

  3. You probably want to manually run VACUUM (or VACUUM FULL for PostgreSQL) after the migration to free up unused rows. This isn't strictly required, but frees up disk space, and removes some of the autovacuum pressure that will run in the background.

  4. Run goatcounter reindex.

  All of this may take a while if you've got a lot of data. For about 500,000 pageviews it takes about 3 minutes on SQLite, but if you've got millions of pageviews it may take an hour or more.

  If you want to keep pageviews while this is running you can:

  1. Write it to a logfile from a proxy or temporary HTTP server and run goatcounter import on this after the migrations are done.

  2. Use goatcounter buffer.

• goatcounter migrate is now goatcounter db migrate. It also behaves a bit different:

  • goatcounter db migrate pending lists only pending migrations, and will use exit code 1 if there are any pending migrations.
  • goatcounter db migrate list lists all migrations, always exits with 0.

• The none value got removed from the -tls flag; use tls=http to not serve TLS. This was confusingly named as you can do -tls=none,acme to still generate ACME certificates, but none implies that nothing is done.

• goatcounter create is now goatcounter db site create, and some flags got changed:

  • -domain is now -vhost.
  • -parent is now -link.
  • -email is now -user.email.
  • -password is now -user.password.

• The -port flag for goatcounter serve is renamed to -public-port. This should clarify that this isn't the listen port, but just the port GoatCounter is publicly accessible on.

• The -site flag got removed from goatcounter import; you can now only use -url to set a GoatCounter site to import to. The automagic API key creation was more confusing than anything else.

  You can use goatcounter db create apitoken to create an API key from the CLI.

• If you build from source, the build flag to set the version changed from:

  -ldflags="-X main.version=..."
  

  to:

  -ldflags="-X zgo.at/goatcounter.Version=..."
  

• The CSV export format was increased to 2; it now includes the parsed browser and system values in addition to the User-Agent header. Version 2.0 will not be able to import the older exports from version 1.

Other changes:

• You can read pageviews from logfiles with the goatcounter import command; you can also send pageviews to goatcounter.com with this (you don’t need to self-host it). See goatcounter help import and the site code documentation for details.

• You can now create multiple users; before there was always a single one. You can add users in Settings → Users.

  As a consequence, "Site settings" and "User preferences" are now split in to two screens. The Settings button in the top-right now displays only site settings, and clicking on your email address in the top right displays user preferences, which every user can configure to their liking.

• You can now configure what's displayed on the dashboard, in what order, and configure some aspects of various "widgets". You can set it in User preferences → Dashboard. Some settings from the main settings page have moved there.

• You can save a default view for the dashboard. Instead of always loading the last week by default, you can now configure it to load the last month, or view by day, or anything you want really.

• You can choose which data to collect; you can disable collecting any User-Agent, location, Referrer information.

• Ability to record state/province/district in addition to country, so it records "US-TX" or "NL-NB" instead of "United States" or "Netherlands".

  This option can be disabled separately from recording the country (enabled by default) and you can set which countries to record it for (defaults to US, RU, CH).

  This requires specifying the path to a GeoIP City database, which isn't included since it's ~30M.

• There are now stable count.v*.js scripts that can use subresource integrity. See the integration code for a list and hashes.

• You can use data-goatcounter-settings on the <script> tag to load the settings (requires count.v2.js or newer).

• New goatcounter buffer command; this allows buffering of pageviews in case the backend is down, running migrations, etc. See goatcounter help buffer for more information.

• The database for PostgreSQL is now created automatically; you no longer need to do this manually.

• You can copy settings from a site to other sites in Settings → Sites.

• Add goatcounter db command; you can now edit and delete sites, users, and API keys from the CLI. The create and migrate commands are now merged in to this as subcommands.

• Add a gcbench utility for inserting random pageviews in a database; for testing and comparing performance. This might be useful for end-users too in some cases, for example to see how much performance difference SQLite and PostgreSQL will give you, or to test if frobbing with server settings makes a difference:

  $ go run ./cmd/gcbench -db sqlite://db/gcbench.sqlite3 -ndays=90 -npaths=100 -nhits=1_000_000
  $ go run ./cmd/gcbench -db postgresql://dbname=gcbench -ndays=90 -npaths=100 -nhits=1_000_000
  

  Right now it doesn't try super-hard to simulate read-world usage patterns: the distribution is always uniform, but it still gives a reasonably accurate indication for comparison purposes.

• Many other minor changes and improvements.

v1.4.2

• Add a "visitor counter" image you can add to your website to display the number of visitors, similar to old-style counters back in the ’90s (#398).

• Other than this, it's mostly contains a few minor bugfixes and the like. You can see a list of changes in the git log: v1.4.1...v1.4.2

v1.4.1

A few small updates, fixes, and performance enhancements. Nothing major.

You can see a list of changes in the git log: v1.4.0...v1.4.1

v1.4.0

Major changes:

• Change defaults for -listen (#336)

  The default for the -listen flag changed from localhost:8081 to :443, which is probably a better and less confusing default for most people. There is also some more detailed docs available in goatcounter help listen.

• Set Cache-Control header for static files (#348)

  The Cache-Control header is now set for static files. Since the "cache busting" happens based on the goatcounter version it's now recommended to set this if you're compiling GoatCounter yourself. See the updated README for instructions.

• Add multi-factor auth (#306)

  TOTP-based multi-factor auth is now supported.

• Better export, export API, add import feature (#316, #318, #329)

  You can now import the CSV exports, useful for migrating from self-hosted to goatcounter.com or vice versa, or for migrating from other systems. There is a web interface and a goatcounter import command.

  The export now supports a "pagination cursor", so you can export only rows you didn't previously export. This is especially useful with the new export API. which should make it easy to sync GoatCounter data with another external platform.

  See http://goatcounter.com/api for details on the export API.

• API for sending pageviews (#357)

  Doing that with the regular /count is actually quite painful, as you quickly run in to ratelimits, need to set specific headers, etc. Adding an API endpoint for that makes things much easier.

• API for creating and editing additional sites (#361)

• Some redesigns (#324, #315, #321 #320)

  The "Totals" is now placed below the Pages; I think it makes more sense there. The Y-axis for the totals is now also independent. There's also been a quite a few restylings.

• Add "text view" mode (#359)

  View your data as a simple table without too much graphics; only the main "Pages" overview is implemented for now.

• Make it easier to skip your own views (#290)

  Previously this required adding custom code, but now loading any page with #toggle-goatcounter added will enable/disable the GoatCounter tracking for that browser.

• Can now manage "additional sites" from self-hosted GoatCounter (#363)

  This wasn't possible before for no other reason than laziness on my part 🙃

• public/count.js is now ISC licensed (#309)

  Previously the EUPL applied, which is fairly restrictive and may prevent people from including/self-hosting the count.js script.

• Add goatcounter db command

  This is mostly useful for writing deploy scripts: goatcounter db schema-sqlite prints the SQLite schema, schema-pgsql prints the PostgreSQL schema, and goatcounter db test tests if the database exists.

• Session hashes are no longer persisted to the database

  This is kind of an internal change, but session hashes are now stored in memory only and never recorded to the database. There's no real reason to persistently store this information, and this is a (small) privacy/GDPR compliance improvement.

v1.3.2

This is a small fix to set some server http timeouts; this is especially relevant if you run GoatCounter "directly" internet-facing, without using a proxy such as HAPRoxy, Varnish, etc.

When using GoatCounter directly internet-facing it's liable to keep connections around for far too long, exhausting the max. number of open file descriptors, especially with "idle" HTTP/2 connections which, unlike HTTP/1.1 Keep-Alive don't have an explicit timeout.

This isn't much of a problem if you're using a proxy in front of it, as most will have some timeouts set by default (unlike Go, which has no timeouts at all by default).

For the backend interface, keeping a long timeout makes sense; it reduces overhead on requests (TLS setup alone can be >200ms), but for the /count request we typically want a much shorter timeout.

Unfortunately, configuring timeouts per-endpoint isn't really supported at this point, although some possible workarounds are mentioned in 1, it's all pretty ugly. We can add "Connection: close" to just close the connection, which is probably much better for almost all cases than keeping a connection open since most people only visit a single page, and keeping a connection open in the off-chance they click somewhere again probably isn't really worth it.

And setting any timeout is better than setting no timeout at all!

v1.3.1

Fixes for SQLite concurrency issues

In some busy workloads various SQLite operations would fail due to "locked tables"; most of this is related to the session code:

• On every pageview a new session is either read (OK) or written (LOCK!)
• At the same time, the cron may be writing data to the db (LOCK!)

On smaller instances this isn't much of an issue since everything is fast enough to not lock for too long, but on longer instances this can be a problem.

Setting SetMaxOpenConns(1) solves this by limiting the connections writing to the database.

Also set the default journal mode to WAL, which should give better performance. Both of this is done in the zgo.at/zdb update.

Add "goatcounter help db" to document database usage a bit better.