\[\cos\frac{5\pi}{24}\cos\frac{\pi}{8} - \sin\frac{\pi}{8}\sin\frac{\pi}{24} =\]
\[= \frac{1}{4} + \frac{1}{2}\cos\frac{\pi}{12} - \frac{1}{2}\cos\frac{\pi}{12} + \frac{\sqrt{3}}{4} =\]
\[= \frac{1}{4} + \frac{\sqrt{3}}{4} = \frac{1 + \sqrt{3}}{4}\]