imported>Bauwens: Migrated current public revision from wiki.cs.hse.ru

2023-09-04T16:22:11Z

Migrated current public revision from wiki.cs.hse.ru

Новая страница

== General Information ==

Lectures: Friday 16h20 -- 17h40, [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens], [https://www.hse.ru/staff/mkaledin Maxim Kaledin], room M202 and on [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]

Seminars: Saturday 14h40 -- 16h00, [https://www.hse.ru/org/persons/225526439 Artur Goldman], room M202 and on zoom (the link will be in telegram)

To discuss the materials, join the [https://t.me/+G0VKOE2-nnkwNDE0 telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2021 last year].

== Problems exam ==

December 21, 13h-16h, computer room G403 ([https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoomlink] for students abroad) 
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book 
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)

== Course materials ==

{| class="wikitable"
|-
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions
|-
|
|| ''Part 1. Online learning''
|-
| 02 Sept
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms [https://drive.google.com/drive/folders/1NXiLbhmO2Ml7jFmnLtjqhOgCoHg7yn9T?usp=sharing movies]
|| [https://www.dropbox.com/s/ryvpnfqfrwyurjc/01slides.pdf?dl=0 sl01]
|| [https://www.dropbox.com/s/oncvg4mxulbt56d/00book_intro.pdf?dl=0 ch00] [https://www.dropbox.com/s/i9pc4kf0zsdeksb/01book_onlineMistakeBound.pdf?dl=0 ch01]
|| [https://www.dropbox.com/s/ztk3n9s5c0vuzd9/01sem.pdf?dl=0 list 1] update 05.09
|| [https://www.dropbox.com/s/r528uroi60gow08/01sol.pdf?dl=0 solutions 1]
|-
| [https://drive.google.com/file/d/16OoCqhh16BKQzyF-HM8RozigyJ3BBVxA/view?usp=sharing 09 Sept]
|| The perceptron algorithm. The standard optimal algorithm.
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02]
|| [https://www.dropbox.com/s/p3auugqwc89132b/02book_sequentialOptimalAlgorithm.pdf?dl=0 ch02] [https://www.dropbox.com/s/b00dcqk1rob7rdz/03book_perceptron.pdf?dl=0 ch03]
|| [https://www.dropbox.com/s/88jgjvxo16zfrjs/02sem.pdf?dl=0 list 2] update 25.09
|| [https://www.dropbox.com/s/pqblktfky8to5hr/02sol.pdf?dl=0 solutions 2]
|-
| [https://www.youtube.com/watch?v=xgyPvnDkyZs 16 Sept]
|| Kernels and the kernel perceptron algorithm. Prediction with expert advice. Recap probability theory.
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]
|| [https://www.dropbox.com/s/3vtxvs4esnvbhlb/04book_predictionWithExperts.pdf?dl=0 ch04] [https://www.dropbox.com/s/l11afq1d0qn6za7/05book_introProbability.pdf?dl=0 ch05]
|| [https://www.dropbox.com/s/fnx2cl5wsgjbmel/03sem.pdf?dl=0 list 3]
|| [https://www.dropbox.com/s/ysg3nipuzryzqoc/03sol.pdf?dl=0 solutions 3]
|-
|
|| ''Part 2. Distribution independent risk bounds''
|-
| [https://www.youtube.com/watch?v=IatjLyN3dRk 23 Sept]
|| Sample complexity in the realizable setting, simple examples and bounds using VC-dimension
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06]
|| [https://www.dropbox.com/s/u5dtpm69bu52fpn/04sem.pdf?dl=0 list 4]
|| [https://www.dropbox.com/s/yurv5s42w3kw5vv/04sol.pdf?dl=0 solutions 4]
|-
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 30 Sept]
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/s/m7xe7k39qzmzapv/08book_VCdimension.pdf?dl=0 ch08]
|| [https://www.dropbox.com/s/u1vpi28gwf0zig4/05sem.pdf?dl=0 list 5]
|| [https://www.dropbox.com/s/3sq7yzv7v4l9tbb/05sol.pdf?dl=0 solutions 5]
|-
| [https://drive.google.com/file/d/17zynIg_CZ6cCNBig5QXmBx7VFS8peyuU/view?usp=sharing 07 Oct]
|| Risk decomposition and the fundamental theorem of statistical learning theory
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]
|| [https://www.dropbox.com/s/8c87619ewkyod4f/09book_riskBounds.pdf?dl=0 ch09]
|| [https://www.dropbox.com/s/eyfczsuwz60moj7/06sem.pdf?dl=0 list 6]
|| [https://www.dropbox.com/s/1te4fzlwj72v6ph/06sol.pdf?dl=0 solutions 6]
|-
| [https://www.youtube.com/watch?v=yMsUH1brAs8 14 Oct]
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma, [https://www.dropbox.com/s/8uravgo5shas55g/07quiz.pdf?dl=0 quiz]
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]
|| [https://www.dropbox.com/s/fg4seoqjbeb7a5g/10book_measureConcentration.pdf?dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]
|| [https://www.dropbox.com/s/qofutar8qy5y53i/07sem.pdf?dl=0 list 7] update 15.10
|| [https://www.dropbox.com/s/w1ud2okt120vm5j/07sol.pdf?dl=0 solutions 7]
|-
|
|| ''Part 3. Margin risk bounds with applications''
|-
| [https://drive.google.com/file/d/1L-BeDxhoHcoDrdlVTlfoMFwnWXKV46cr/view?usp=sharing 21 Oct]
|| Simple regression, support vector machines, margin risk bounds, and neural nets
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/s/jaym44fmif2uw05/13book_SVM.pdf?dl=0 ch13]
|| [https://www.dropbox.com/s/bzo6msrxcfa8tpp/08sem.pdf?dl=0 list 8]
|| [https://www.dropbox.com/s/pe7yctcr93yaw95/08sol.pdf?dl=0 solutions 8]
|-
| [https://youtu.be/9FhFxLHR4eE 04 Nov]
|| Kernels: RKHS, representer theorem, risk bounds
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]
|| [https://www.dropbox.com/s/ply602zthd7r3jv/14book_kernels.pdf?dl=0 ch14]
|| [https://www.dropbox.com/s/54xdufimavhd646/09sem.pdf?dl=0 list 9]
|| [https://www.dropbox.com/s/i3rx26ya6kvm5p2/09sol.pdf?dl=0 solutions 9]
|-
| [https://youtu.be/1oUXZy6Sqlk 11 Nov]
|| AdaBoost and the margin hypothesis
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]
|| [https://www.dropbox.com/s/e7m1cs7e8ulibsf/15book_AdaBoost.pdf?dl=0 ch15]
|| [https://www.dropbox.com/s/nu4a55qbfqlp3bl/10sem.pdf?dl=0 list 10]
|| [https://www.dropbox.com/s/t64mjapdzcm1313/10sol.pdf?dl=0 solutions 10]
|-
| [https://youtu.be/GL574ljefJ8 18 Nov]
|| Implicit regularization of stochastic gradient descent in neural nets
||
|| [https://www.dropbox.com/s/b4xac5uki7l1ysq/16book_implicitRegularization.pdf?dl=0 ch16]
|| no seminar
||
|-
|
|| ''Part 4. Other topics''
|-
| [https://youtu.be/kOXi_m9dBzE 25 Nov]
|| Regression I: fixed design with sub-Gaussian noise
||
|| [https://disk.yandex.ru/i/tI7NiGsvQP0Jww notes12]
|| [https://disk.yandex.ru/d/9fFxVlMw4kPfEQ list 12]
|| [https://disk.yandex.ru/i/5vBE2VC7zNC3Rg solutions 12]
|-
| [https://youtu.be/GEYT_IxXEX0 02 Dec]
|| Multiarmed bandids I
||
|| [https://disk.yandex.ru/i/lvqXofEbaFkfAA notes13]
|| [https://disk.yandex.ru/i/ZXXJbBiJUPNiOw list 13]
||
|-
| [https://youtu.be/Uybf6mCp2Es 09 Dec]
|| Multiarmed bandids II (optional)
||
|| [https://disk.yandex.ru/i/Nqy9-wmZ-g5o8g notes14]
|| [https://disk.yandex.ru/d/0Qupo2CNSjS_pQ notebook], [https://disk.yandex.ru/d/suY6d58SFf09Bg notebook(solved)]
||
|-
| 16 Dec
|| ''Colloquium''
|-
|}

The lectures in October and November are based on the book:
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. This book can be downloaded from [https://libgen.is Library Genesis] (the link changes sometimes and sometimes vpn is needed).



== Grading formula ==

Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.

All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point.

There is no rounding except on the final grade. Grades fractional grades above 5/10 are rounded up, those below 5/10 are rounded down.

Autogrades: if you only need 4/10 to pass with maximal final score, it will be given automatically. This may happen because of extra questions and bonuses from quizzes.

For students who want to pass with 4/10 with minimal effort: each year on the exam, I ask to calculate the VC-dimension or Rademacher complexity of some class. It should be easy to have 4/10 for the final exam. If you understand all lecture notes, you pass the colloquium with maximal score. Together this is enough. If only a few students fail and the grades are at least 3.8/10 then failed students may resubmit a few homework tasks to pull up the grade. (This happened in the last 3 years.)

== Colloquium ==

[https://www.dropbox.com/s/7djya6nc8ietd32/colloqQuest.pdf?dl=0 Rules and questions.] Update 12/12 added question 24 and corrected typos.



== Homeworks ==

Email to brbauwens-at-gmail.com. Start the subject line with SLT-HW.

Deadline before the start of the lecture, every other lecture.

Sat. 17 Sept 18h10: problems 1.7, 1.8, 2.9, and 2.11 
Sat. 01 Oct 18h10: see lists 3 and 4, and 2.10 
Fri. 14 Oct 16h20: see problem lists 5 and 6 
Sat. 05 Nov 20h00: see problem lists 7 and 8 
Sat. 19 Nov 20h00: see problem lists 9 and 10 
Sun. 04 Dec 23h59: see problem list 12 send it to maxkaledin@gmail.com with subject line SLT-HW-Reg <YourName>_<YourSurname>
 

== Office hours ==

{| class="wikitable"
|-
! Person !! Monday !! Tuesday !! Wednesday !! Thursday !! Friday !!
|-
| Bruno Bauwens || 15-20h || || || || 18-20h ||
|-
| Maxim Kaledin || Write || in || Telegram || time is || flexible ||
|-
|}

It is always good to send an email in advance. Questions and feedback are welcome.

Statistical learning theory 2022 - История изменений

imported>Bauwens: Migrated current public revision from wiki.cs.hse.ru