TMPGEnc 4.0 XPress & Pulldown - Part II

[acoustiking]

A BILLION thanks to Propellerhead on the other TMPGEnc thread for playing around with TMPGEnc and posting his findings as well!!! I Gave this a whirl too, and basically decided that TEMPEnc 4.0 is the "be all end all" for the HV20 24p mess for me. It did EXACTLY what I wanted it to do. Basically follow 'Propellerhead's' set up instructions, summed up as (I like when I have numbered steps for my pea brain, so sorry if you don't, just scroll)

1. Capture (I'm a Vegas guy)
2. Import into TMPGEnc
3. Click "Filters" button
4. Make sure "Deinterlace" is checked
5. Change Deinterlace Mode to "24fps special animation"
6. Change Deinterlace Method to "Inverse Pulldown"
7. Click "OK"
8. Click "Format" button
9. Chose "HDV format MPEG file" (double click it)
10. Click "MPEG output" in 'non-standard settings' box on the right
11. Click "yes" when the warning box comes up (who reads those anyway?)
12. Change framerate to '23.976'
13. Change DC Component to '9 bit'
14. Change Display Mode to 'progressive'
15. Save as template if you'd like
16. Go to Encode and encode, of course

Output file looks great. All properties read HDV .m2t at 23.976

I import back into Vegas and editing goes great

To check further, I downloaded the Premier CS3 trial (thank god for high speed internet...bandwidth hog Adobe programs sure are!) and imported the new file into that (I used the HDV 1080p30 settings, and just did a little property changing). Upon importing the new file/playback, there was a goofy red full frame screen for the first few frames, but it just pixelates out quick and played fine. I have played with AE7 a while back and I assume "rendering on the timeline" refers to having to play the file first to get it to play back "smoothly". This new file played back at real time, and didn't seem to be rendering while playing. Here's some links for your own testing to see if I'm backasswards on my assumptions I've been doing this right. I know it's an exta step of encoding, but it's fast, it's a batch render, you can apply other effects, all for only for just under a $100.

Raw 24p HDV with TV mode on & shutter speed at 1/48
http://hv20.info/yopu/TMPGenc 60i-24p start.m2t

Encoded TMPGEnc with Pulldown completed
http://hv20.info/yopu/TMPGEnc 24p after.m2t

Please tell me this is all the way it's supposed to be as I'm sick of screwing around with this pulldown fiasco...

[jkatme]

hey acousti
How did you change the cs3 project settings?
How did you turn hdv1080p30 into 24?

Nice work if it removes the pulldown properly.
cheers

[acoustiking]

How did you change the cs3 project settings?
How did you turn hdv1080p30 into 24?

More numbered lists (luv'em)

1. New Project
2. Click HDV 1080p30 first
3. Then click 'custom settings' tab
4. Under 'general' hit drop box named "timebase" and change to 23.976
5. When you do #4 above, a couple other properties change automatically
6. Save your new template before you click "OK"
7. Nest time you start with a new project, it will be under "custom" templates

Here a .swf screen capture of the setup:
http://hv20.info/yopu/HV20 Premier Setup.swf

You'll notice the red frames at the end when I play the clip. Mind you the stutter in the clip playback is because the screen capture is done at about 15fps

Here's a link to a free swf player if you don't have one already...
http://www.globfx.com/products/swfplayer/

[pascalbrown]

thanks for the numbered points, makes following instructions so much easier. I'll give this a blast when I get home now that I've got the software. Currently I'm far away from my main box though so will report back next week.

[acoustiking]

Nice work if it removes the pulldown properly.

Here's some good backup that pulldown is being done right. Also, this is good to see for those who are still a little blurry why pulldown is needed, and what the camera is actually doing when it is filling in the ~5 frames each second when recording 24p to a 60i format.

Clip recorded in 24p mode. More fun with screen capture! (in .swf format again, use the above link in the original post for a free player if you don't have one already)

Frame by frame advancement of the raw footage in the editing timeline captured from the HV20 in Vegas. Notice the interlaced "ghosting" when the camera is filling the frames.
http://hv20.info/yopu/TMPGEnc.60i-24p.FxF.swf

Frame by frame advancement of the same clip with pulldown performed by TMPGEnc in the editing timeline imported back into Vegas. The pulldown was done correctly, and if you downloaded the clip in the original post, you'll see that it's smooth as well.
http://hv20.info/yopu/TMPGEnc.24p.encoded.FxF.swf

[Farnsworth]

One question before I give this a try...

Will it batch process? When I have a whole tape full of clips, I don't want to process them one at a time.

[acoustiking]

Will it batch process?

Yes. I just add the whole folder with the clips, from scene detection, and go through all the steps.

I just did a batch of 54 clips equalling about 40m of footage, and for some reason two clips were strange during playback (slow motiony and choppy). I just reran the two clips and will go to town on my editing soon. Of course it takes a bit to load them all, and the processing takes some time...nothing crazy. Correctness & automation is my attraction to this. Plus for a $100, the program does ALOT more to top it off. It has pretty simple RGB, YUV, etc color correctors in which you use sliders to "correct" the clip(s). I had some older footage shot under florescents that was yellowish, and I batch ran the clips, "fixing" them all. Now When I can find the time to put it together, color correcting is done.

[Kilgor270]

Good Work!


I also revisted this app and after careful analysis I concur it does work!. I was wrong in the Part 1 of this thread. My inital problem was playing back the newly created clip on the timeline in CS3. Initially it appeared as if it needed rendered again by Premiere. I now know that that was caused by me and not TMPGE.

My last detail to check is file size. Can anyone compare the original capture vs the processed clip?

Thanks

Kilgor

[acoustiking]

My last detail to check is file size. Can anyone compare the original capture vs the processed clip?

Misc 1:38 clip I just processed:

original 317mb
clip with pulldown 315mb

[pascalbrown]

Misc 1:38 clip I just processed:

original 317mb
clip with pulldown 315mb

Can you confirm what the input clip was (m2t?) and what codec/settings you used, because this is the first time I've seen a smaller file size after pulldown! Thank you.

[pc001]

Is this possible with TMPGENC 3 Express?

Perry

[pascalbrown]

I've just tried this and can confirm it appears to me (disclaimer; I know nothing) to be working. Thanks for the numbered steps

What I'm most confused/perplexed about is that my input file size is almost identical to my output file size! While this might seem to make sense given it is only taking out certain parts of the data, it doesn't correspond with using the hv20pulldown.exe program. Is TMPGENC giving me a lossless .m2t file (I followed the instructions in the first post)? Or am I introducing my first generation of loss by using this method? Help please.

[propellerhead]

Thanks validating this acoustiking.

Yes folks, it is really that simple. However, I'm not so certain that rerendering to an m2t file can be considered "lossless". It's prolly pretty damn good but I suspect you're going to see some issues with chroma sampling.

[Luthyr]

I have been using this to downconvert to 24P DVD MPEGs, and it has been working excellently. It's very easy to use and produces great results.

[Noel]

Got my new HV20 3days ago. 24P is the way to go. The recordings blew me away. My computer can't capture using HDVSplit though. Don't know why. But I'm willing the try new softwares. With your 1st step. Do you capture using Vegas or TMPGEnc. I'm new to the whole software editing. But the 24P recording is very motivating.

[Anx]

I just purchased TMPGEnc and gone through the steps detailed above, and compared the results to the HV20pulldown.exe method, which I had been using previously. I actually used the source clip provided by the Eugenia's HV20pulldown tutorial (which is great for getting it running, btw), launched two instances of MediaPlayerClassic, and compared frames as best I could between the two conversions. I noticed the follwing things between the conversions:

1. The HV20Pulldown version was much blockier - if you go to around 4 seconds in, and look at the shadow of the car driving through the frame, TMPGEnc version looks much smoother and less pixelated.
2. In the same shot, the TMPGEnc image looks slightly washed out - the blacks in the HV20Pulldown look darker.
3. This may be related to issue 2, but there also appears to be more detail in the rims - but at the same time, the detail on the stop sign looks about the same.

Either way, I did this more to get a feel for whether or not I would be comfortable running all my video through another generation of compression, and what the effect would be. For me, the convienience of this workflow overrides the issues I spotted, so it's well worth the $99 price tag in my book. I also tried to get Cineform NeoHDV going on my system, but for whatever reason I could never get anything to play back any of it's files correctly.

[BobbyMurcerFan]

Has anyone been able to successfully compare TMPGEnc to CineForm?

Thanks.

[BobbyMurcerFan]

... I noticed the follwing things between the conversions:

1. The HV20Pulldown version was much blockier - if you go to around 4 seconds in, and look at the shadow of the car driving through the frame, TMPGEnc version looks much smoother and less pixelated.
2. In the same shot, the TMPGEnc image looks slightly washed out - the blacks in the HV20Pulldown look darker.
3. This may be related to issue 2, but there also appears to be more detail in the rims - but at the same time, the detail on the stop sign looks about the same.

...

Anx,

What codecs and formats did you use when creating the HV20Pulldown and TMPGEnc files?

I would imagine the differences you are seeing are a result of codec and file format choices and not the actual pulldown removal process itself.

[Anx]

With the HV20Pulldown, I was using Lagarith, so lossless... I was following everything basically word for word from Eugenia's blog post (I would have been screwed without it, she did a great job), and I used her test footage for my comparisons.

On TMPGEnc I used exactly what was detailed at the beginning of this thread, so it was being recompressed into HDV... There will be some loss due to the generation, but I couldn't see much apart from what I detailed above. Of course, the files are roughly the same size as the original, which is a bonus as opposed to Lagarith which were substancially larger, to say the least (and is to be expected).

It's certainly possible that the codec is the contributing factor, but for my money, the recompressed version looked better (except for point 2/3 which are probably related) - point 1 should have favored Lagarith..

[BobbyMurcerFan]

From what I understand, the compression part of a codec can vary from software to software (or camera to camera) but decompression has to be identical across implementations.

So programs may compress MPEG-2 differently, but they will all decompress MPEG-2 exactly the same way.

Using this piece of information, HV20Pulldown and TMPGEnc should decompress the same HDV footage to identical files. The next step, removing the pulldown itself doesn't seem like any mean feat, its determining which frames to remove and deleting them.

Once the pulldown frames are removed, both methods should still have identical footage, assuming they both chose the correct frames to delete.

So it's when the footage is recompressed that I believe the difference is introduced. Using MPEG-2 again shouldn't be a problem because all the frames were at one time defined by MPEG-2. While it's true that some frames have been removed and MPEG-2 is both intra and inter frame compressed, the codec shouldn't have that hard a time figuring out where to start a new I frame (intra compressed only) and the subsequent interdependent P and B frames.

So what is HV20Pulldown doing during recompression? (Or maybe it adds an intermediate step between frame removal and recompression that affects the footage to be recompressed?) I have no idea. But it sounds like there is a glitch somewhere.

I realize that Lagarith is lossless, but perhaps trying a different codec for recompression may give a better result. But if Lagarith is being implemented properly, then IMHO, the footage it's being implemented upon has been somehow altered after the pulldown frames were removed.

(Of course I could be totally wrong, LOL, but I believe the above makes sense.)

BTW, thanks very much for all the work you've done on this topic. It helps us all .

[von_Rodg]

I certainly like the TMPGEnc 4 Xpress inverse telecine method for its directness and small output file size - which can be played smoothly in realtime.

Can anyone confirm to what extent this method is lossless (barring the junk interlaced frames that are deliberately cut).

I guess I'm worried about the HDV MPEG re-encoding at the end. Does this encoding add an extra generation of data loss, compared with the Farnesworth / SSudzick / Eugenia method which encodes to a lossless codec?

i.e. does the process: capture--->HDV (loss#1)-->24p pulldown--->HDV MPEG (loss#2?) produce a second round of loss with re-HDV MPEG encoding, or has this exact same data been lost already at first HDV encoding and therefore the second encoding is 'free' / lossless?

Thank you anyone that can help with this

BTW...I've tried both methods, and once setup the HV20pulldown.exe method is also very simple, but I find it difficult to compare the quality of end results on my PC screen and don't have a BluRay burner / Playstation etc yet to send a HiDef 24p output to HDTV.

[BobbyMurcerFan]

Do you have to re-encode into HDV with TMPGenc? I thought there was a variety of output formats available?

http://tmpgenc.pegasys-inc.com/en/pr...e4xp_spec.html

[von_Rodg]

BobbyFan, you are correct. I can encode to Lagarith AVI from TMPEnc.

However, oddly, using the same Lagarith settings (eg YUY2) and AVI video settings (1440x1080, 23.976 progressive) the HV20pulldown.exe method gives a 288Mb .AVI file with 24 bit video sample size (on XP file properties tab) while TMPGEnc gives a smaller 240Mb .AVI with 16 bit video sample size (and with correct 1.333 pixel ratio...unlike the HV20pulldown method).

My best guess is that this difference in video sample size reflects the colour space used leading in to AVI coding and this is increased by VirtualDubMod "Full processing mode" video setting? Not sure this actually leads to any color enhancement given that the source HDV colour depth is limiting and the fact that in both instances Lagarith settings are for YUY2.

In Windows medai player both AVIs appear similar, for what that's worth.

[von_Rodg]

Going back to my original question, does anyone know if re-encoding to HDV MPEG in TMPGEnc as outlined at the top of this thread, after the inverse telecine (ie 24p reverse pulldown) process from HDV, is lossless or lossy?

The files are smaller and play in real time so it would be a nice way to edit but I don't want to introduce another generation of artefact just for that.

Thanks for your comments

[BobbyMurcerFan]

BobbyFan, you are correct. I can encode to Lagarith AVI from TMPEnc.

However, oddly, using the same Lagarith settings (eg YUY2) and AVI video settings (1440x1080, 23.976 progressive) the HV20pulldown.exe method gives a 288Mb .AVI file with 24 bit video sample size (on XP file properties tab) while TMPGEnc gives a smaller 240Mb .AVI with 16 bit video sample size (and with correct 1.333 pixel ratio...unlike the HV20pulldown method).

My best guess is that this difference in video sample size reflects the colour space used leading in to AVI coding and this is increased by VirtualDubMod "Full processing mode" video setting? Not sure this actually leads to any color enhancement given that the source HDV colour depth is limiting and the fact that in both instances Lagarith settings are for YUY2.

In Windows medai player both AVIs appear similar, for what that's worth.

Well it seems like HV20Pulldown is creating an AVI with 24bit sound and TMPGenc is creating one with 16bit sound. I'm assuming the sound is uncompressed in both cases, as it is in DV (as opposed to HDV). Perhaps this accounts for the file size difference?

As for MPEG-2 encoding being lossy after you remove the pulldown, yes it will be. It has to be b/c MPEG-2 is a lossy compression and decompression methodology, and MPEG-2 uses previous and (in some cases) subsequent frames to define the current frame. By deleting frames, MPEG-2 will have to re-encode which means loss.

BUT I believe the loss will be pretty minimal b/c all of the frames were defined by MPEG-2 before. If you were adding effects, I would say you'd experience more of a problem with loss, but these frames have already been "cleaned" of what MPEG-2 can't encode. (At least that's the way I understand it.)

What format are you going to be working with in your NLE?