186 lines
9.6 KiB
Markdown
186 lines
9.6 KiB
Markdown
# zxcvbn frequency lists
|
||
|
||
The two word lists in this directory are unmodified copies of third-party
|
||
frequency lists. They are read only at build time: the rules in
|
||
`Telegram/cmake/lib_zxcvbn.cmake` transliterate them and compile the result
|
||
into the `zxcvbn-c` password dictionary behind
|
||
`Telegram/SourceFiles/ui/passcode_strength.cpp`. No built artifact contains
|
||
these files: they are build-time inputs to a generated header. They are
|
||
tracked, so every source distribution of Telegram Desktop ships them
|
||
verbatim, and the attribution below covers those copies as well as the
|
||
dictionary compiled from them.
|
||
|
||
## Provenance
|
||
|
||
- Title: *FrequencyWords*
|
||
- Creator: Hermit Dave
|
||
- Repository: <https://github.com/hermitdave/FrequencyWords>
|
||
- Upstream commit: `f8a65e6ddc17e0baa2e366a909986798d8dbe55b` (2019-02-15,
|
||
"Additional languages including pl, pt and zh variants", the last upstream
|
||
commit touching either file)
|
||
- Derivation: generated by the upstream author from the OpenSubtitles2018
|
||
corpus (<http://opus.nlpl.eu/OpenSubtitles2018.php>); one `word count`
|
||
pair per line, separated by a single space, in descending order of count;
|
||
50,000 lines each, all lowercase, LF line endings, no BOM.
|
||
|
||
| Tracked file | Upstream path | Bytes | Lines |
|
||
| --- | --- | --- | --- |
|
||
| `ru_50k.txt` | `content/2018/ru/ru_50k.txt` | 998,861 | 50,000 |
|
||
| `uk_50k.txt` | `content/2018/uk/uk_50k.txt` | 887,259 | 50,000 |
|
||
|
||
sha256 digests, equal to the GitHub raw files at the pinned commit (verified
|
||
2026-09-06):
|
||
|
||
```
|
||
6095f507cc167488ec66ada5a85ac50433503a08ad24a07c6eabdf54352c4e7f ru_50k.txt
|
||
881686fe66a168b79f6a9ee2f6cbbf944cfb12bfb7040913ef535cb29bd86ec9 uk_50k.txt
|
||
```
|
||
|
||
`.gitattributes` pins `Telegram/Resources/zxcvbn/*.txt` to `text eol=lf`, so
|
||
the digests hold on a Windows checkout as well.
|
||
|
||
## Licence
|
||
|
||
Upstream's `README.md` states, under "License", at the upstream head
|
||
`525f9b560de45753a5ea01069454e72e9aa541c6` (2022-02-07) — head on
|
||
2026-09-06, the day these copies were taken:
|
||
|
||
> MIT License for code.
|
||
> CC-by-sa-4.0 for content.
|
||
|
||
The licence *version* in that statement has changed exactly once. At the
|
||
pinned commit `f8a65e6ddc17e0baa2e366a909986798d8dbe55b` (2019-02-15, the
|
||
only commit that ever touched either list) the section read
|
||
`MIT License for codes.<br>` / `CC-by-sa-3.0 for contents.`; commit
|
||
`0c6a7e40cfd8c645f9e91ff70cd94d7aa8c85c09` (2019-02-15) reworded it to
|
||
`MIT License for code.<br>` / `CC-by-sa-3.0 for content.`, leaving the
|
||
version at 3.0; and commit `34bb6447a96d4434dc2d4b24e0cf946c523090b3`
|
||
(2020-08-11, "Update README.md") replaced `CC-by-sa-3.0` with
|
||
`CC-by-sa-4.0` in that one line, since when the section has read as quoted
|
||
above.
|
||
|
||
The word lists are content, not code, and the licence that governs these
|
||
copies is the Creative Commons Attribution-ShareAlike 4.0 International
|
||
licence (CC BY-SA 4.0), <https://creativecommons.org/licenses/by-sa/4.0/>,
|
||
through the licensor's standing offer at head — not through the pin, and
|
||
not merely because the bytes are unchanged. A public licence is an offer
|
||
made when the material is made available, and a licensor may add a new offer
|
||
over unchanged files without withdrawing the old one (CC BY-SA 3.0 §7(b),
|
||
CC BY-SA 4.0 §6(c)). The pin fixes which bytes are tracked, and the digests
|
||
above stay pinned to it; it does not fix which offer they were taken under.
|
||
|
||
The earlier 3.0 offer is perpetual and remains available, but it is not
|
||
relied on here. Were the 4.0 offer ever disputed, 3.0 alone would still
|
||
reach CC BY-SA 4.0, because 3.0 §4(b)(ii) permits distributing an Adaptation
|
||
under "a later version of this License with the same License Elements"; it
|
||
would not reach the GNU GPL directly, because Creative Commons records that
|
||
"Currently, no non-CC licenses have been designated as compatible with BY-SA
|
||
3.0".
|
||
|
||
The repository's `LICENSE` file (MIT, "Copyright (c) 2016 Hermit Dave")
|
||
covers the generator code only. Upstream supplies no separate copyright or
|
||
disclaimer notice for the content beyond the licence line quoted above.
|
||
|
||
## Changes made
|
||
|
||
The lists are used as build-time input only: the word column is kept, each
|
||
word is transliterated to Latin letters by `zxcvbn_translit`, words that are
|
||
still non-ASCII or shorter than two letters are dropped, four OpenSubtitles
|
||
subtitle-markup strings (`chffffff`, `fntahoma`, `b1`, `i0`) are dropped,
|
||
and the result is compiled into the zxcvbn dictionary; the tracked files are
|
||
unmodified copies.
|
||
The count column only orders the words: the generated dictionary keeps a
|
||
word's rank, not its count.
|
||
|
||
## Compatibility with the application licence
|
||
|
||
Telegram Desktop is distributed under the GNU General Public License, version
|
||
3 or (at your option) any later version (`/LICENSE`, `/LEGAL`), with the
|
||
OpenSSL linking exception. CC BY-SA 4.0 content may be adapted into a GPLv3
|
||
project on this basis:
|
||
|
||
- Creative Commons, "Compatible Licenses",
|
||
<https://creativecommons.org/share-your-work/licensing-considerations/compatible-licenses/>:
|
||
"GPLv3: The GNU General Public License version 3 was declared a
|
||
'BY-SA–Compatible License' for version 4.0 …" and "Note that compatibility
|
||
with the GPLv3 is one-way only, which means you may license your
|
||
contributions to adaptations of BY-SA 4.0 materials under GPLv3, but you
|
||
may not license your contributions to adaptations of GPLv3 projects under
|
||
BY-SA 4.0." (declared 2015-10-08).
|
||
- Creative Commons wiki, "ShareAlike compatibility: GPLv3",
|
||
<https://wiki.creativecommons.org/wiki/ShareAlike_compatibility:_GPLv3>:
|
||
"When someone creates an adaptation of a BY-SA licensed work and includes
|
||
it in a GPLv3-licensed project, both licenses apply and downstream users
|
||
must comply with both licenses. However, Section 2(a)(5)(B) of BY-SA 4.0
|
||
allows anyone who receives the adapted material downstream to satisfy the
|
||
conditions of both BY-SA and GPLv3 (i.e. attribution and ShareAlike) in
|
||
the manner dictated by the GPLv3." and "this compatibility determination
|
||
only allows movement one way, from BY-SA 4.0 to GPLv3."
|
||
- Free Software Foundation, "Various Licenses and Comments about Them",
|
||
<https://www.gnu.org/licenses/license-list.html#ccbysa>: "CC BY-SA 4.0 is
|
||
one-way compatible with the GNU GPL version 3: this means you may license
|
||
your modified versions of CC BY-SA 4.0 materials under GNU GPL version 3,
|
||
but you may not relicense GPL 3 licensed works under CC BY-SA 4.0. Because
|
||
Creative Commons lists only version 3 of the GNU GPL on its compatible
|
||
licenses list, it means that you can not license your adapted CC BY-SA
|
||
works under the terms of 'GNU GPL version 3, or (at your option) any later
|
||
version.' However, Section 14 of the GNU GPL version 3 allows licensors to
|
||
specify a proxy to determine whether future versions of the GNU GPL can be
|
||
used. Therefore, if someone adapts a CC BY-SA 4.0 work and incorporates it
|
||
into a GNU GPL version 3 licensed project, they can specify Creative
|
||
Commons as their proxy …"
|
||
|
||
**Consequence for this repository — an owner decision this file records but
|
||
cannot make.** Creative Commons declared only GPL version 3 a
|
||
BY-SA-Compatible License, and the wiki page quoted above instructs adapters:
|
||
"You must specify version 3 of the GPL when redistributing a GPL project
|
||
into which BY-SA 4.0 content is adapted." Telegram Desktop's own offer is
|
||
broader: `/LEGAL` — and every source file header that points at it — says
|
||
"either version 3 of the License, or (at your option) any later version".
|
||
Once the dictionary compiled from this corpus is linked in, that broader
|
||
offer cannot be made for the combined work unless the Program itself names a
|
||
proxy under GPL §14 ("If the Program specifies that a proxy can decide which
|
||
future versions of the GNU General Public License can be used…"). This file
|
||
is not the Program's licence statement and cannot make that designation. The
|
||
repository owner has to pick one:
|
||
|
||
- (a) add a GPL §14 proxy designation naming Creative Commons
|
||
(<https://creativecommons.org/compatiblelicenses>) to `/LEGAL`, after
|
||
which "or any later version" works again for the combined work as and when
|
||
Creative Commons declares a later GPL version compatible;
|
||
- (b) accept that builds containing this dictionary are offered under GPL
|
||
version 3 only, and qualify `/LEGAL` accordingly;
|
||
- (c) drop this corpus — no BY-SA content, no constraint.
|
||
|
||
Until (a) or (b) is recorded in `/LEGAL`, this component and the tree's "or
|
||
any later version" offer are in tension. The in-app statement,
|
||
`lng_about_text2`, already reads "licensed under the GNU GPL version 3". The
|
||
tracked lists themselves remain verbatim copies under CC BY-SA 4.0.
|
||
|
||
## Reproducing the dictionary
|
||
|
||
The build performs these steps itself; run them by hand from the checkout
|
||
root after a Debug build to check that the generated header is reproducible.
|
||
`<dir>` is any scratch directory outside the checkout.
|
||
|
||
```
|
||
out/Telegram/Debug/zxcvbn_translit \
|
||
Telegram/Resources/zxcvbn/ru_50k.txt <dir>/ru_50k-latin.txt
|
||
out/Telegram/Debug/zxcvbn_translit \
|
||
Telegram/Resources/zxcvbn/uk_50k.txt <dir>/uk_50k-latin.txt
|
||
out/Telegram/Debug/zxcvbn_dictgen -o <dir>/dict-src.h \
|
||
Telegram/ThirdParty/zxcvbn/words-eng_wiki.txt \
|
||
Telegram/ThirdParty/zxcvbn/words-female.txt \
|
||
Telegram/ThirdParty/zxcvbn/words-male.txt \
|
||
Telegram/ThirdParty/zxcvbn/words-passwd.txt \
|
||
Telegram/ThirdParty/zxcvbn/words-surname.txt \
|
||
Telegram/ThirdParty/zxcvbn/words-tv_film.txt \
|
||
<dir>/ru_50k-latin.txt \
|
||
<dir>/uk_50k-latin.txt
|
||
cmp <dir>/dict-src.h out/Telegram/zxcvbn_gen/dict-src.h
|
||
```
|
||
|
||
The input order matters: the generator interleaves its inputs by rank, so
|
||
the six upstream lists come first, in the order `lib_zxcvbn.cmake` lists
|
||
them, followed by the Russian and then the Ukrainian intermediate. `cmp`
|
||
must report no difference.
|