Post

New Caesar

We found a brand new type of encryption, can you break the secret code? (Wrap with picoCTF{}) fegdeogdgecoeocgcgchcfcffccfca

New Caesar

image-1619

Download the Python code.

Let’s walk through the encryption step by step.

1. The custom “base16” encoding

1
ALPHABET = "abcdefghijklmnop"

Instead of using hexadecimal digits 0-9a-f, the code uses:

1
2
3
4
5
0 -> a
1 -> b
2 -> c
...
15 -> p

The function:

1
2
3
4
5
6
7
def b16_encode(plain):
    enc = ""
    for c in plain:
        binary = "{0:08b}".format(ord(c))
        enc += ALPHABET[int(binary[:4], 2)]
        enc += ALPHABET[int(binary[4:], 2)]
    return enc

takes each character and:

  1. Converts it to an 8-bit binary value.

  2. Splits it into two 4-bit halves (nibbles).

  3. Converts each nibble to a letter a-p.

Example: 'A'

ASCII:

1
A = 65

Binary:

1
01000001

Split:

1
0100 0001

Convert each nibble:

1
2
0100 = 4 = e
0001 = 1 = b

So:

1
'A' -> "eb"

2. The shift function

1
2
3
4
def shift(c, k):
    t1 = ord(c) - ord('a')
    t2 = ord(k) - ord('a')
    return ALPHABET[(t1 + t2) % 16]

This is a Vigenère/Caesar shift on the alphabet a-p.

Example with key 'd':

1
2
3
4
5
6
c = e  (4)
k = d  (3)

(4 + 3) mod 16 = 7

result = h

So:

1
shift('e','d') = 'h'

3. Encryption process

1
2
3
4
b16 = b16_encode(flag)

for i, c in enumerate(b16):
    enc += shift(c, key[i % len(key)])

Because:

1
assert len(key) == 1

the same key character is used for every position.

So encryption is:

1
2
3
4
5
6
7
flag
 ↓
custom base16 (a-p)
 ↓
Caesar shift by key value
 ↓
ciphertext

Example

Suppose:

1
2
flag = "A"
key = "c"

Step 1: b16 encode

1
2
3
A -> 01000001
   -> 0100 0001
   -> e b

Result:

1
"eb"

Step 2: shift by key c

1
c = 2

Shift each character:

1
2
e(4) + 2 = g
b(1) + 2 = d

Ciphertext:

1
"gd"

Why decryption is easy

Since:

1
len(key) == 1

there are only 16 possible keys:

1
a b c d e f g h i j k l m n o p

Trying all 16 keys is trivial. For each key:

  1. Reverse the shift:
1
(original - key) mod 16
  1. Convert pairs of letters back to bytes.

That’s why this scheme is very weak cryptographically—it is effectively a Caesar cipher over a 16-character alphabet after a reversible encoding step.

The decryptor simply reverses the two operations performed during encryption, in reverse order.

Encryption recap

The original code does:

1
2
3
4
5
6
7
8
9
Plaintext
   ↓
b16_encode()
   ↓
letters a-p
   ↓
shift by key
   ↓
Ciphertext

So decryption must do:

1
2
3
4
5
6
7
8
9
Ciphertext
   ↓
reverse shift
   ↓
letters a-p
   ↓
b16_decode()
   ↓
Plaintext

Step 1: Reverse the shift

Encryption used:

1
2
def shift(c, k):
    return ALPHABET[(t1 + t2) % 16]

where:

1
2
t1 = value of plaintext character
t2 = value of key character

For example:

1
2
3
4
5
e (4)
key = c (2)

(4 + 2) mod 16 = 6
→ g

To recover the original character, subtract instead of add:

1
2
3
4
def unshift(c, k):
    t1 = ord(c) - ord('a')
    t2 = ord(k) - ord('a')
    return ALPHABET[(t1 - t2) % 16]

Example:

1
2
3
4
5
cipher = g (6)
key    = c (2)

(6 - 2) mod 16 = 4
→ e

So:

1
g → e

Step 2: Reverse the custom base16 encoding

Recall that:

1
2
3
4
5
a = 0
b = 1
c = 2
...
p = 15

During encryption, each byte became two letters.

Example:

1
'A'

ASCII:

1
65

Binary:

1
01000001

Split:

1
0100 0001

Convert:

1
2
0100 → 4 → e
0001 → 1 → b

Result:

1
"A" → "eb"

To decode:

1
2
3
4
high = ALPHABET.index('e')   # 4
low  = ALPHABET.index('b')   # 1

byte = (high << 4) | low

Calculation:

1
2
3
4
5
(4 << 4) | 1
=
64 + 1
=
65

ASCII 65:

1
'A'

Recovered successfully.


The b16 decoder

1
2
3
4
5
6
7
8
9
10
11
def b16_decode(enc):
    out = ""

    for i in range(0, len(enc), 2):
        high = ALPHABET.index(enc[i])
        low = ALPHABET.index(enc[i + 1])

        byte = (high << 4) | low
        out += chr(byte)

    return out

What happens?

Suppose:

1
enc = "eb"

Loop:

1
2
high = 4
low = 1

Rebuild byte:

1
byte = (4 << 4) | 1

Result:

1
65

Convert back:

1
chr(65)

Result:

1
'A'

Complete decryptor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
import string

  

LOWERCASE_OFFSET = ord("a")

ALPHABET = string.ascii_lowercase[:16] # abcdefghijklmnop

  

def unshift(c, k):

t1 = ord(c) - LOWERCASE_OFFSET

t2 = ord(k) - LOWERCASE_OFFSET

return ALPHABET[(t1 - t2) % len(ALPHABET)]

  

def b16_decode(enc):

out = ""

for i in range(0, len(enc), 2):

high = ALPHABET.index(enc[i])

low = ALPHABET.index(enc[i + 1])

  

byte = (high << 4) | low

out += chr(byte)

  

return out

  

def decrypt(ciphertext, key):

# Reverse Vigenère shift

b16 = ""

for i, c in enumerate(ciphertext):

b16 += unshift(c, key[i % len(key)])

  

# Decode base16-like encoding

return b16_decode(b16)

  
  

# Example usage

ciphertext = "fegdeogdgecoeocgcgchcfcffccfca"

for key in ALPHABET:

try:

print(f"{key}: {decrypt(ciphertext, key)}")

except:

pass

  

print(decrypt(ciphertext, key))

Why brute force works

The original program contains:

1
assert len(key) == 1

So the key is only one character from:

1
abcdefghijklmnop

That’s only:

1
16 possible keys

Therefore you can simply try them all:

1
2
for key in ALPHABET:
    print(key, decrypt(ciphertext, key))

Look for readable English or a recognizable flag among the outputs to identify the recovered plaintext.

Flag: picoCTF{et_tu?_77866c61}


This post is licensed under CC BY 4.0 by the author.